Click to follow the official account, Python dry goods will be delivered in time👇
Source丨Network
1. Introduction to python operation database
The Python standard database interface is Python DB-API, which provides developers with a database application programming interface. The Python database interface supports a very large number of databases, you can choose the database that suits your project:
-
GadFly -
mSQL -
MySQL -
PostgreSQL -
Microsoft SQL Server 2000 -
Informix -
Interbase -
Oracle -
Sybase...
You can visit the Python Database Interface and API for a detailed list of supported databases.
You need to download different DB API modules for different databases. For example, if you need to access Oracle database and Mysql data, you need to download Oracle and MySQL database modules.
DB-API is a specification. It defines a series of necessary objects and database access methods in order to provide a consistent access interface for a variety of underlying database systems and a variety of database interface programs.
Python's DB-API implements interfaces for most databases. After using it to connect to each database, you can operate each database in the same way.
Python DB-API usage process:
-
Introduce API modules. -
Get a connection to the database. -
Execute SQL statements and stored procedures. -
Close the database connection.
2. python operates MySQL module
Python operates MySQL mainly in two ways:
-
DB module (native SQL) -
PyMySQL (support python2.x/3.x) -
MySQLdb (currently only supports python2.x) -
ORM framework -
SQLAchemy
2.1 PyMySQL module
This article mainly introduces the PyMySQL module, and MySQLdb is used in a similar way
2.1.1 Install PyMySQL
PyMySQL is a MySQL driver written in Python that allows us to manipulate MySQL databases in Python.
pip install PyMySQL
2.2 Basic usage
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
# 创建连接
conn = pymysql.connect(host="127.0.0.1", port=3306, user='zff', passwd='zff123', db='zff', charset='utf8mb4')
# 创建游标(查询数据返回为元组格式)
# cursor = conn.cursor()
# 创建游标(查询数据返回为字典格式)
cursor = conn.cursor(pymysql.cursors.DictCursor)
# 1. 执行SQL,返回受影响的行数
effect_row1 = cursor.execute("select * from USER")
# 2. 执行SQL,返回受影响的行数,一次插入多行数据
effect_row2 = cursor.executemany("insert into USER (NAME) values(%s)", [("jack"), ("boom"), ("lucy")]) # 3
# 查询所有数据,返回数据为元组格式
result = cursor.fetchall()
# 增/删/改均需要进行commit提交,进行保存
conn.commit()
# 关闭游标
cursor.close()
# 关闭连接
conn.close()
print(result)
"""
[{'id': 6, 'name': 'boom'}, {'id': 5, 'name': 'jack'}, {'id': 7, 'name': 'lucy'}, {'id': 4, 'name': 'tome'}, {'id': 3, 'name': 'zff'}, {'id': 1, 'name': 'zhaofengfeng'}, {'id': 2, 'name': 'zhaofengfeng02'}]
"""
2.3 Get the newly created data auto-increment ID
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
# 创建连接
conn = pymysql.connect(host="127.0.0.1", port=3306, user='zff', passwd='zff123', db='zff', charset='utf8mb4')
# 创建游标(查询数据返回为元组格式)
cursor = conn.cursor()
# 获取新创建数据自增ID
effect_row = cursor.executemany("insert into USER (NAME)values(%s)", [("eric")])
# 增删改均需要进行commit提交
conn.commit()
# 关闭游标
cursor.close()
# 关闭连接
conn.close()
new_id = cursor.lastrowid
print(new_id)
"""
8
"""
2.4 Query operation
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
# 创建连接
conn = pymysql.connect(host="127.0.0.1", port=3306, user='zff', passwd='zff123', db='zff', charset='utf8mb4')
# 创建游标
cursor = conn.cursor()
cursor.execute("select * from USER")
# 获取第一行数据
row_1 = cursor.fetchone()
# 获取前n行数据
row_2 = cursor.fetchmany(3)
#
# # 获取所有数据
row_3 = cursor.fetchall()
# 关闭游标
cursor.close()
# 关闭连接
conn.close()
print(row_1)
print(row_2)
print(row_3)
⚠️ In order to fetch data, you can use cursor.scroll(num,mode) to move the cursor position, such as:
-
cursor.scroll(1,mode='relative') # move relative to the current position -
cursor.scroll(2,mode='absolute') # Move relative to absolute position
2.5 Prevent SQL Injection
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
# 创建连接
conn = pymysql.connect(host="127.0.0.1", port=3306, user='zff', passwd='zff123', db='zff', charset='utf8mb4')
# 创建游标
cursor = conn.cursor()
# 存在sql注入情况(不要用格式化字符串的方式拼接SQL)
sql = "insert into USER (NAME) values('%s')" % ('zhangsan',)
effect_row = cursor.execute(sql)
# 正确方式一
# execute函数接受一个元组/列表作为SQL参数,元素个数只能有1个
sql = "insert into USER (NAME) values(%s)"
effect_row1 = cursor.execute(sql, ['wang6'])
effect_row2 = cursor.execute(sql, ('wang7',))
# 正确方式二
sql = "insert into USER (NAME) values(%(name)s)"
effect_row1 = cursor.execute(sql, {'name': 'wudalang'})
# 写入插入多行数据
effect_row2 = cursor.executemany("insert into USER (NAME) values(%s)", [('ermazi'), ('dianxiaoer')])
# 提交
conn.commit()
# 关闭游标
cursor.close()
# 关闭连接
conn.close()
In this way, SQL operations are more secure. If you need more detailed documentation, please refer to the PyMySQL documentation. However, it seems that the implementation of these SQL databases is not the same. The parameter placeholders of PyMySQL use C formatters such as %s, while the placeholders of the sqlite3 module that comes with Python seem to be question marks (?). So read the documentation carefully when using other databases. Welcome to PyMySQL's documentation
3. Database connection pool
There is a problem with the above method. It can be satisfied in the case of single thread. The program needs to create and release connections frequently to complete the operation of the database. Then, what problems will our program/script cause in the case of multi-threading? At this time, We need to use the database connection pool to solve this problem!
3.1 DBUtils module
DBUtils is a Python module for implementing database connection pooling.
This connection pool has two connection modes:
-
Create a connection for each thread. Even if the thread calls the close method, it will not close, but just put the connection back into the connection pool for its own thread to use again. The connection is automatically closed when the thread terminates -
Create a batch of connections to the connection pool for shared use by all threads (recommended)
3.2 Mode 1
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
from DBUtils.PersistentDB import PersistentDB
import pymysql
POOL = PersistentDB(
creator=pymysql, # 使用链接数据库的模块
maxusage=None, # 一个链接最多被重复使用的次数，None表示无限制
setsession=[], # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]
ping=0,
# ping MySQL服务端，检查是否服务可用。# 如：0 = None = never, 1 = default = whenever it is requested, 2 = when a cursor is created, 4 = when a query is executed, 7 = always
closeable=False,
# 如果为False时， conn.close() 实际上被忽略，供下次使用，再线程关闭时，才会自动关闭链接。如果为True时， conn.close()则关闭链接，那么再次调用pool.connection时就会报错，因为已经真的关闭了连接（pool.steady_connection()可以获取一个新的链接）
threadlocal=None, # 本线程独享值得对象，用于保存链接对象，如果链接对象被重置
host='127.0.0.1',
port=3306,
user='zff',
password='zff123',
database='zff',
charset='utf8',
)
def func():
conn = POOL.connection(shareable=False)
cursor = conn.cursor()
cursor.execute('select * from USER')
result = cursor.fetchall()
cursor.close()
conn.close()
return result
result = func()
print(result)
3.2 Mode 2
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import time
import pymysql
import threading
from DBUtils.PooledDB import PooledDB, SharedDBConnection
POOL = PooledDB(
creator=pymysql, # 使用链接数据库的模块
maxconnections=6, # 连接池允许的最大连接数，0和None表示不限制连接数
mincached=2, # 初始化时，链接池中至少创建的空闲的链接，0表示不创建
maxcached=5, # 链接池中最多闲置的链接，0和None不限制
maxshared=3,
# 链接池中最多共享的链接数量，0和None表示全部共享。PS: 无用，因为pymysql和MySQLdb等模块的 threadsafety都为1，所有值无论设置为多少，_maxcached永远为0，所以永远是所有链接都共享。
blocking=True, # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错
maxusage=None, # 一个链接最多被重复使用的次数，None表示无限制
setsession=[], # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]
ping=0,
# ping MySQL服务端，检查是否服务可用。# 如：0 = None = never, 1 = default = whenever it is requested, 2 = when a cursor is created, 4 = when a query is executed, 7 = always
host='127.0.0.1',
port=3306,
user='zff',
password='zff123',
database='zff',
charset='utf8'
)
def func():
# 检测当前正在运行连接数的是否小于最大链接数，如果不小于则：等待或报raise TooManyConnections异常
# 否则
# 则优先去初始化时创建的链接中获取链接 SteadyDBConnection。
# 然后将SteadyDBConnection对象封装到PooledDedicatedDBConnection中并返回。
# 如果最开始创建的链接没有链接，则去创建一个SteadyDBConnection对象，再封装到PooledDedicatedDBConnection中并返回。
# 一旦关闭链接后，连接就返回到连接池让后续线程继续使用。
conn = POOL.connection()
# print('连接被拿走了', conn._con)
# print('池子里目前有', POOL._idle_cache, '\r\n')
cursor = conn.cursor()
cursor.execute('select * from USER')
result = cursor.fetchall()
conn.close()
return result
result = func()
print(result)
⚠️ Since the threadsafety value of pymysql, MySQLdb, etc. is 1, the threads in the connection pool of this mode will be shared by all threads, so it is thread-safe. If there is no connection pool, when using pymysql to connect to the database, there is no problem with single-threaded applications, but if it involves multi-threaded applications, it needs to be locked. Once locked, the connection will inevitably wait in line. When there are many requests, the performance will be reduced. will be lowered.
3.3 Lock
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
import threading
from threading import RLock
LOCK = RLock()
CONN = pymysql.connect(host='127.0.0.1',
port=3306,
user='zff',
password='zff123',
database='zff',
charset='utf8')
def task(arg):
with LOCK:
cursor = CONN.cursor()
cursor.execute('select * from USER ')
result = cursor.fetchall()
cursor.close()
print(result)
for i in range(10):
t = threading.Thread(target=task, args=(i,))
t.start()
3.4 No lock (error)
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "shuke"
# Date: 2018/5/13
import pymysql
import threading
CONN = pymysql.connect(host='127.0.0.1',
port=3306,
user='zff',
password='zff123',
database='zff',
charset='utf8')
def task(arg):
cursor = CONN.cursor()
cursor.execute('select * from USER ')
# cursor.execute('select sleep(10)')
result = cursor.fetchall()
cursor.close()
print(result)
for i in range(10):
t = threading.Thread(target=task, args=(i,))
t.start()
At this point, you can view the connection status in the database: show status like 'Threads%';
4. The database connection pool is used in combination with pymsql
# cat sql_helper.py
import pymysql
import threading
from DBUtils.PooledDB import PooledDB, SharedDBConnection
POOL = PooledDB(
creator=pymysql, # 使用链接数据库的模块
maxconnections=20, # 连接池允许的最大连接数，0和None表示不限制连接数
mincached=2, # 初始化时，链接池中至少创建的空闲的链接，0表示不创建
maxcached=5, # 链接池中最多闲置的链接，0和None不限制
#maxshared=3, # 链接池中最多共享的链接数量，0和None表示全部共享。PS: 无用，因为pymysql和MySQLdb等模块的 threadsafety都为1，所有值无论设置为多少，_maxcached永远为0，所以永远是所有链接都共享。
blocking=True, # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错
maxusage=None, # 一个链接最多被重复使用的次数，None表示无限制
setsession=[], # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]
ping=0,
# ping MySQL服务端，检查是否服务可用。# 如：0 = None = never, 1 = default = whenever it is requested, 2 = when a cursor is created, 4 = when a query is executed, 7 = always
host='192.168.11.38',
port=3306,
user='root',
passwd='apNXgF6RDitFtDQx',
db='m2day03db',
charset='utf8'
)
def connect():
# 创建连接
# conn = pymysql.connect(host='192.168.11.38', port=3306, user='root', passwd='apNXgF6RDitFtDQx', db='m2day03db')
conn = POOL.connection()
# 创建游标
cursor = conn.cursor(pymysql.cursors.DictCursor)
return conn,cursor
def close(conn,cursor):
# 关闭游标
cursor.close()
# 关闭连接
conn.close()
def fetch_one(sql,args):
conn,cursor = connect()
# 执行SQL，并返回收影响行数
effect_row = cursor.execute(sql,args)
result = cursor.fetchone()
close(conn,cursor)
return result
def fetch_all(sql,args):
conn, cursor = connect()
# 执行SQL，并返回收影响行数
cursor.execute(sql,args)
result = cursor.fetchall()
close(conn, cursor)
return result
def insert(sql,args):
"""
创建数据
:param sql: 含有占位符的SQL
:return:
"""
conn, cursor = connect()
# 执行SQL，并返回收影响行数
effect_row = cursor.execute(sql,args)
conn.commit()
close(conn, cursor)
def delete(sql,args):
"""
创建数据
:param sql: 含有占位符的SQL
:return:
"""
conn, cursor = connect()
# 执行SQL，并返回收影响行数
effect_row = cursor.execute(sql,args)
conn.commit()
close(conn, cursor)
return effect_row
def update(sql,args):
conn, cursor = connect()
# 执行SQL，并返回收影响行数
effect_row = cursor.execute(sql, args)
conn.commit()
close(conn, cursor)
return effect_row
PS: It can be encapsulated into a class using static methods for easy use
Recommended reading Click on the title to jump 1. 7 lines of code make station B crash for 3 hours
Think this article was helpful to you? please share with more people
It is recommended to pay attention to "Python Workshop" to improve Python skills
Like and watching is the biggest support ❤️