我目前正在使用MySQL和Python从网络上抓取数据。具体来说,我正在抓取表数据并将其插入到数据库中。我当前的解决方案可以工作,但是我觉得它效率极低,如果不重写代码,很可能会锁定数据库。这是我当前使用的(部分代码):
itemBank = [] for row in rows: itemBank.append((tempRow2,tempRow1,tempRow3,tempRow4)) #append data #itemBank List of dictionaries representing data from each row of the table. i.e. ('Item_Name':"Tomatoes",'Item_Price':"10",'Item_In_Stock':"10",'Item_Max':"30") for item in itemBank: tempDict1 = item[0] tempDict2 = item[1] tempDict3 = item[2] tempDict4 = item[3] q = """ INSERT IGNORE INTO TABLE1 ( Item_Name, Item_Price, Item_In_Stock, Item_Max, Observation_Date ) VALUES ( "{0}", "{1}", "{2}", "{3}", "{4}" ) """.format(tempDict1['Item_Name'],tempDict2['Item_Price'],tempDict3['Item_In_Stock'], tempDict4['Item_Max'],getTimeExtra) try: x.execute(q) conn.commit() except: conn.rollback()
执行表的每一行都很麻烦。我尝试使用executemany,但似乎无法弄清楚如何正确访问字典的值。因此,executemany在给定数据结构的情况下,如何在这里使用它插入数据库?
executemany
itemBank = [] for row in rows: itemBank.append(( tempRow2['Item_Name'], tempRow1['Item_Price'], tempRow3['Item_In_Stock'], tempRow4['Item_Max'], getTimeExtra )) #append data q = """ insert ignore into TABLE1 ( Item_Name, Item_Price, Item_In_Stock, Item_Max, Observation_Date ) values (%s,%s,%s,%s,%s) """ try: x.executemany(q, itemBank) conn.commit() except: conn.rollback()
希望对您有帮助