Xevent Reader

Xevent Reader

Xevent Reader is a tool written in Python that reads Extend Event Files of SQL Server and exports parsed results into SQL Server. It is designed to be an alternative to SSMS when the exporting table function crashes. The Reader has two main functions: read and parse xevent files into SQL Server tables and import csv files into SQL Server.

Xevent Reader on Github

Tool Interface
SSMS

Install

  1. Install Microsoft ODBC Driver17 for SQL Server x64
  2. Download the tool. Link
  3. Protection programs (Windows Security) might be triggered, please allow access.
  4. Ready to go.

Usage

  1. Open the tool, please wait for the GUI pops up.

  2. Full execution mode

    1. Function: read and parse xevent files into SQL Server
    2. Press the Select Files button
      1. Select xevent files (*.xel) in the drop-down list.
      2. Multiple xevent files can be added.
      3. Different types of xevent files can be added, results will be written into one table.
    3. Choose authentication mode for SQL Server
      1. Default: Windows Authentication
    4. Fill in the connection info.
      1. HOSTNAME: the server you want to connect to.
      2. DATABASE: the database that holds the result table. If the database does not exist, the Reader would create a new one.
      3. TABLE NAME: the table that holds parsing results. If the table name exists, the original content of that table will be replaced.
      4. USERNAME: the user name for SQL Authentication.
      5. PASSWORD: password of SQL Authentication account.
    5. Press Run: full execution
      1. If any field is missed, there will be warning messages.
      2. Execution progress is visualized in the command line window.
      3. A temporary .csv file will be generated for debugging purposes, you can choose to delete or keep it when the execution is finished.
  3. Importing CSV to SQL Server Only mode

    1. Function: import a formatted csv file into a specified SQL Server

      This mode is designed to import temp csv files to SQL Server generated by the full execution mode when the parsing is completed but importing results to SQL Server is failed.

    2. Press the Select Files button

      1. Select csv files (*.csv) in the drop down list.
      2. Only one CSV file can be added.
    3. Choose authentication mode for SQL Server

      1. Default: Windows Authentication
    4. Fill in the connection info.

      1. HOSTNAME: the server you want to connect to.
      2. DATABASE: the database that holds the result table. If the database does not exist, the Reader would create a new one.
      3. TABLE NAME: the table that holds parsing results. If the table name exists, the original content of that table will be replaced.
      4. USERNAME: the user name for SQL Authentication.
      5. PASSWORD: password of SQL Authentication account.
    5. Press Run: import csvCSV to SQL only

      1. If any field is missed, there will be warning messages.
      2. Execution progress is visualized in the command line window.
      3. You can choose to delete or keep the selected CSV file when the execution is finished.

Known Issue

  1. The GUI gets stuck from time to time when the file selection window is closed.

    Sol: Press 'Enter' in the command line window would work

  2. The GUI window is not popped up when the tool is started for a while.

    Sol: Press 'Enter' in the command line window would work

  3. When dealing with large (total size) xevent files, the importing to SQL Server step might be very slow. The normal importing rate should be several hundred items per second. This might due to the fast_executemany for SQL Server is not triggered.

    Sol:

    Firstly, read and parse xevent files with Run: full execution to get a temporary csv file. Terminate the tool when the message Action: Importing to SQL Server table ... is shown.

    Secondly, reopen the tool, select the temporary CSV file, fill in necessary connection info, and press Importing CSV to SQL Server Only (follow steps of Importing csCSVo SQL Server Only mode)

Development Notes

This tool is my intern project at the Microsoft CSS team, and it is designed to solve the problem that SSMS usually crashes when exporting large xevent files into SQL Server tables.

Version 1

Key Idea

Parse the xevent file and use pandas Dataframe to hold the result, automatically expanding columns to fit all event objects.

Problem

Way to slow

Version 2

Key Idea

Use pandas to read xevent files from SQL Server and parse them with vaex. Apply functions to vaex dataframe columns and also expand columns to fit all event objects.

Problem

Need to convert vaex dataframe into pandas dataframe when exporting results to SQL Server, this step would cause memory error when processing large xevent files. The memory error can be solved by reducing the chuck size of importing data into SQL but the exporting rate would be super slow.

Final Version

Key Idea

Take advantage of csv file features, and replace the step of modifying the dataframe with string operations. The key trick is, that if we remove the "" of a string that is formatted like "A, B, C", we would have a CSV format file. Thus, the time and space used in operating dataframes are largely reduced.

Originally, unique columns of events are found by SQL query, which is a little bit slow.

1
2
3
4
5
select event_data from 
(select *, row_number() over (partition by object_name order by timestamp_utc desc) as rn
from
sys.fn_xe_file_target_read_file('C:\xeventFile.xel',null, null, null)) t
where rn = 1

Then I found that using PD.drop_duplicate is much more faster.

Source Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
"""
Author: Jiacheng Xie
Date: 2022/8/17
"""
from tkinter import Tk,filedialog, messagebox, Button, W,Label,Entry
import pyodbc
from pymssql import connect as connection
from pandas import read_xml, DataFrame,read_sql,concat,read_csv
from tqdm import tqdm
from sqlalchemy import create_engine
from time import time
from os import remove
import gc

# import the following to avoid no 'pymssql' found issue when packing
from pymssql import _mssql
from pymssql import _pymssql
import uuid
import decimal


global authenticationMode
authenticationMode = 0
global files
files = ()

"""
Find column names in the xml field of xevent files
"""
def findColumns(xml_pretty_str):
return ' '.join(read_xml(xml_pretty_str)['name'].tolist())

"""
Visualize pd.to_sql, add a process bar
"""
def chunker(seq, size):
# from http://stackoverflow.com/a/434328
return (seq[pos:pos + size] for pos in range(0, len(seq), size))

def insert_with_progress(df,conn, tableName):
#conn = create_engine(DB_URI2, execution_options=dict(stream_results=True),fast_executemany=True)

chunksize = int(len(df) / 10) # 10%
if chunksize == 0:
chunksize = 1

with tqdm(total=len(df)) as pbar:
for i, cdf in enumerate(chunker(df, chunksize)):
replace = "replace" if i == 0 else "append"
cdf.to_sql(con=conn, name=tableName, if_exists=replace, index=False)
pbar.update(chunksize)

"""
This is the core function of the Xevent reader.
It parses xml and edits the result to get correct values for each column.
Return a comma separated string of all column values.

"""
def xmlToString2(xml_pretty_str, column_list_ordered):

df_xml = read_xml(xml_pretty_str)#.set_index('name')

names = list(df_xml['name'].values)

# chain of if...else to deal with columns with xml data
if 'input_relation' in names:

pattern = '<data name="input_relation">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'input_relation','value'] = data

elif 'calculator' in names:

pattern = '<data name="calculator">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'calculator','value'] = data

elif 'stats_collection' in names:

pattern = '<data name="stats_collection">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'stats_collection','value'] = data

elif 'showplan_xml' in names:

pattern = '<data name="showplan_xml">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'showplan_xml','value'] = data

elif 'output_parameters' in names:

pattern = '<data name="output_parameters">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'output_parameters','value'] = data

elif 'data' in names:

pattern = '<data name="data">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'data','value'] = data

elif 'execution_stats_report' in names:

pattern = '<data name="execution_stats_report">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'execution_stats_report','value'] = data

elif 'xml_report' in names:

pattern = '<data name="xml_report">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'xml_report','value'] = data

elif 'xml_report_filtered' in names:

pattern = '<data name="xml_report_filtered">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'xml_report_filtered','value'] = data

elif 'server_memory_grants' in names:

pattern = '<data name="server_memory_grants">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'server_memory_grants','value'] = data

elif 'blocked_process' in names:

pattern = '<data name="blocked_process">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'blocked_process','value'] = data

else:

pattern = '<data name="blocked_process_filtered">'
data = xml_pretty_str[xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'):xml_pretty_str.find('</value>', xml_pretty_str.find('<value>', xml_pretty_str.find(pattern)+len(pattern))+len('<value>'))]
df_xml.loc[df_xml['name'] == 'blocked_process_filtered','value'] = data
# exclude
if 'data_stream' in names:
df_xml.drop(df_xml[df_xml['name'] == 'data_stream'].index, inplace = True)
names.remove('data_stream')

if 'timestamp' in names:
df_xml.drop(df_xml[df_xml['name'] == 'timestamp'].index, inplace = True)
names.remove('timestamp')

if 'text' in df_xml:
df_xml['value'] = df_xml['text'].fillna(df_xml['value'])

#df_xml['value'] = df_xml['value'].apply(lambda x: str(x).replace(',','^'))


values = list(df_xml['value'].values)


result = []

for name in column_list_ordered:
if name in names:
result.append(str(values[names.index(name)]).replace(',','^'))
else:
result.append('None')

return ','.join(result).replace('\t','').replace('\n','')


def setWinAuthentication():
global authenticationMode
authenticationMode = 0
print('Windows Authentication Selected')
print('Specify HOSTNAME, DATABASE, TABLE NAME:')

def setSQLAuthentication():
global authenticationMode
authenticationMode = 1
print("SQL Authentication Selected")
print('Specify HOSTNAME, DATABASE, TABLE NAME, USERNAME, PASSWORD')

def selectFiles():
global files
files = filedialog.askopenfilenames(filetypes=[('xevent files', '.xel'),('csv files', '.csv')])
print('Files selected:')
print(files)
print('Select Authentication Mode:')

"""
The key idea is to save execution result into a csv file and then remove the "" to simulate the
process of creating columns. Since the editted file has the same format as a csv file, when reading
it again into a pandas dataframe would result in an expanded table with all columns of xevent names.
"""
def run():
if len(files) == 0:
messagebox.askokcancel(title = 'Message',message='No file selected!')
return 'No file selected!'

HOSTNAME = entryHOSTNAME.get()
DATABASE = entryDATABASE.get()
TableName = entryTableName.get()
USERNAME = entryUSERNAME.get()
PASSWORD = entryPASSWORD.get()
if authenticationMode == 1 and (len(USERNAME) == 0 or len(PASSWORD) == 0):
messagebox.askokcancel(title = 'Message',message='Please specify USERNAME and PASSWORD for SQL Authentication')
return 'SQL Authentication Failed'
if len(HOSTNAME) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify HOSTNAME')
return 'Invalid input'
if len(DATABASE) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify DATABASE')
return 'Invalid input'
if len(TableName) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify Table Name')
return 'Invalid input'


# First connection, connect to master by default
if authenticationMode == 0:
print('Action: Connecting to ' + 'master' + " <Windows Authentication> ...")
else:
print('Action: Connecting to ' + 'master' + " <SQL Authentication>")
DATABASE_master = 'master'
DRIVER = 'ODBC+Driver+17+for+SQL+Server'
DB_URI_SQL = 'mssql+pyodbc://{}:{}@{}/{}?driver={}'.format(
USERNAME, PASSWORD, HOSTNAME, DATABASE_master,DRIVER
)
DB_URI_Win = 'mssql+pyodbc://{}/{}?trusted_connection=yes&driver={}'.format(
HOSTNAME, DATABASE_master,DRIVER
)

DB_URI2_Win = 'mssql+pyodbc://{}/{}?trusted_connection=yes&driver={}'.format(
HOSTNAME, DATABASE,DRIVER
)
DB_URI2_SQL = 'mssql+pyodbc://{}:{}@{}/{}?driver={}'.format(
USERNAME, PASSWORD, HOSTNAME, DATABASE,DRIVER
)

if authenticationMode == 0:
DB_URI = DB_URI_Win
DB_URI2 = DB_URI2_Win
conn = create_engine(DB_URI, execution_options=dict(stream_results=True),fast_executemany=True)
connect = connection(host=HOSTNAME,
database=DATABASE_master,
autocommit=True)

else:
DB_URI = DB_URI_SQL
DB_URI2 = DB_URI2_SQL
conn = create_engine(DB_URI, execution_options=dict(stream_results=True),fast_executemany=True)
connect = connection(host=HOSTNAME,
database=DATABASE_master,
user=USERNAME,
password=PASSWORD,
autocommit=True)
cursor = connect.cursor()
print("... Connected ...")

# create new database if not exist
new_database = str("'" + DATABASE + "'")

cursor.execute("select * From master.dbo.sysdatabases where name=" + new_database)
DBresult = cursor.fetchall()

if len(DBresult) == 0: # if DB not exist
cursor.execute("CREATE DATABASE " + new_database.strip("'"))
print('Message: ' + new_database.strip("'") + " created")
else:
print('Message: ' + new_database.strip("'") + " exist")
connect.close()

# set table name
print('Message: result will be written into '+ TableName + ' ...')
print('Message: table with the same name will be repalced ...')

# second connection, same Authentication as first
if authenticationMode == 0:
print('Action: Connecting to ' + new_database.strip("'") + " <Windows Authentication> ...")
else:
print('Action: Connecting to ' + 'master' + " <SQL Authentication>")

conn = create_engine(DB_URI2, execution_options=dict(stream_results=True),fast_executemany=True)

print("... Connected ...")
# read from sql
print('Action: Read files ...')

df = DataFrame()
for file in tqdm(files):
file = str("'" + file + "'").replace('/',"\\")
query = "select * from sys.fn_xe_file_target_read_file(" + file + ",null, null, null)"
temp_df = read_sql(query, conn)
df = concat([df,temp_df],ignore_index=True)
print("Message:")
print(df.info())
df.rename(columns={'object_name':'event_name'}, inplace = True)

df_col = df.drop_duplicates(['event_name']).copy()
df_col['columns'] = df_col['event_data'].apply(findColumns)
column_list_string = df_col['columns'].values.tolist()
column_list_set = set(' '.join(column_list_string).split())
column_list_set.discard('timestamp')
#column_list_set.discard('object_name')
column_list_set.discard('data_stream')
column_list_ordered = list(column_list_set)
column_list_ordered.sort()
column_list_string = str(column_list_ordered).replace('[','').replace(']','').replace("'",'').replace(" ",'')
print('Message: the following columns will be written ...')
print(column_list_ordered)

# Parsing
tqdm.pandas(desc='pandas bar')
print("Action: Parsing xevent ...")
df[column_list_string] = df['event_data'].progress_apply(xmlToString2,column_list_ordered=column_list_ordered)

df2 = DataFrame(df, columns = ['event_name','timestamp_utc',column_list_string])

del df
gc.collect()

# save result to csv
print('Action: Save and edit temp csv file ...')
temp_name = str(int(time()))
df2.to_csv('./'+ temp_name + '.csv',index = False)

del df2
gc.collect()

with open('./'+ temp_name + '.csv','r',encoding="utf-8") as f:
lines = f.readlines()
with open('./'+ temp_name + '.csv', 'w',encoding="utf-8") as f_w:
for line in tqdm(lines):
line = line.replace('"','')#.replace('[','').replace(']','')
f_w.write(line)

# read from editted csv file
print("Action: Reading csv file ...")
chunks2 = read_csv('./'+ temp_name + '.csv',dtype=str,chunksize=2000)

df_done = DataFrame()
for chunk in tqdm(chunks2):
df_done = concat([df_done, chunk],ignore_index=True)
print('Message: ')
print(df_done.info())

# importing to SQL Server

print('Action: Importing to SQL Server table ...')
insert_with_progress(df_done,conn, TableName)
print('Job done!')

del df_done
gc.collect()

delTemp = messagebox.askyesno(title = 'Message',message='Job Done! Delete temp csv file?')
if delTemp:
remove('./'+ temp_name + '.csv')
print("Press Enter to Continue ...")


"""
Importing csv files into SQL Server.
"""
def run_csv():
if len(files) == 0:
messagebox.askokcancel(title = 'Message',message='No file selected!')
return 'No file selected!'

HOSTNAME = entryHOSTNAME.get()
DATABASE = entryDATABASE.get()
TableName = entryTableName.get()
USERNAME = entryUSERNAME.get()
PASSWORD = entryPASSWORD.get()
if authenticationMode == 1 and (len(USERNAME) == 0 or len(PASSWORD) == 0):
messagebox.askokcancel(title = 'Message',message='Please specify USERNAME and PASSWORD for SQL Authentication')
return 'SQL Authentication Failed'
if len(HOSTNAME) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify HOSTNAME')
return 'Invalid input'
if len(DATABASE) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify DATABASE')
return 'Invalid input'
if len(TableName) == 0:
messagebox.askokcancel(title = 'Message',message='Please specify Table Name')
return 'Invalid input'


# First connection, connect to master by default
if authenticationMode == 0:
print('Action: Connecting to ' + 'master' + " <Windows Authentication> ...")
else:
print('Action: Connecting to ' + 'master' + " <SQL Authentication>")
DATABASE_master = 'master'
DRIVER = 'ODBC+Driver+17+for+SQL+Server'
DB_URI_SQL = 'mssql+pyodbc://{}:{}@{}/{}?driver={}'.format(
USERNAME, PASSWORD, HOSTNAME, DATABASE_master,DRIVER
)
DB_URI_Win = 'mssql+pyodbc://{}/{}?trusted_connection=yes&driver={}'.format(
HOSTNAME, DATABASE_master,DRIVER
)

DB_URI2_Win = 'mssql+pyodbc://{}/{}?trusted_connection=yes&driver={}'.format(
HOSTNAME, DATABASE,DRIVER
)
DB_URI2_SQL = 'mssql+pyodbc://{}:{}@{}/{}?driver={}'.format(
USERNAME, PASSWORD, HOSTNAME, DATABASE,DRIVER
)

if authenticationMode == 0:
DB_URI = DB_URI_Win
DB_URI2 = DB_URI2_Win
conn = create_engine(DB_URI, execution_options=dict(stream_results=True),fast_executemany=True)
connect = connection(host=HOSTNAME,
database=DATABASE_master,
autocommit=True)

else:
DB_URI = DB_URI_SQL
DB_URI2 = DB_URI2_SQL
conn = create_engine(DB_URI, execution_options=dict(stream_results=True),fast_executemany=True)
connect = connection(host=HOSTNAME,
database=DATABASE_master,
user=USERNAME,
password=PASSWORD,
autocommit=True)
cursor = connect.cursor()
print("... Connected ...")

# create new database if not exist
new_database = str("'" + DATABASE + "'")

cursor.execute("select * From master.dbo.sysdatabases where name=" + new_database)
DBresult = cursor.fetchall()

if len(DBresult) == 0: # if DB not exist
cursor.execute("CREATE DATABASE " + new_database.strip("'"))
print('Message: ' + new_database.strip("'") + " created")
else:
print('Message: ' + new_database.strip("'") + " exist")
connect.close()

# set table name
print('Message: result will be written into '+ TableName + ' ...')
print('Message: table with the same name will be repalced ...')

# second connection, same Authentication as first
if authenticationMode == 0:
print('Action: Connecting to ' + new_database.strip("'") + " <Windows Authentication> ...")
else:
print('Action: Connecting to ' + 'master' + " <SQL Authentication>")

conn = create_engine(DB_URI2, execution_options=dict(stream_results=True),fast_executemany=True)

print("... Connected ...")

# read from editted csv file
print("Action: Reading csv file ...")
chunks2 = read_csv(files[0],dtype=str,chunksize=2000)

df_done = DataFrame()
for chunk in tqdm(chunks2):
df_done = concat([df_done, chunk],ignore_index=True)
print('Message: ')
print(df_done.info())

# importing to SQL Server

print('Action: Importing to SQL Server table ...')
insert_with_progress(df_done,conn, TableName)
print('Job done!')

del df_done
gc.collect()

delTemp = messagebox.askyesno(title = 'Message',message='Job Done! Delete temp csv file?')
if delTemp:
remove(files[0])
print("Press Enter to Continue ...")

print('Select Xevent Files:')
myWindow = Tk()

myWindow.title('Xevent Reader')
myWindow.resizable(0, 0)

b0=Button(myWindow, text='Select Files', relief='ridge', width=45, height=2, command = selectFiles)
b0.grid(row=0, columnspan=2, sticky=W, padx=5,pady=5)

b1=Button(myWindow, text='SQL Authentication', relief='ridge', width=20, height=2, command = setSQLAuthentication)
b1.grid(row=1, column=0, sticky=W, padx=5,pady=5)
b2=Button(myWindow, text='Windows Authentication',relief='ridge',width=20, height=2, command = setWinAuthentication)
b2.grid(row=1, column=1, sticky=W, padx=5, pady=5)


Label(myWindow, text="HOSTNAME").grid(row=2)
Label(myWindow, text="DATABASE").grid(row=3)
Label(myWindow, text="TABLE NAME").grid(row=4)
Label(myWindow, text="Skip Following Blanks If Windows Authentication Is Selected",relief = 'groove',bd = 4).grid(row=5,columnspan=2)
Label(myWindow, text="USERNAME").grid(row=6)
Label(myWindow, text="PASSWORD").grid(row=7)

entryHOSTNAME=Entry(myWindow)
entryDATABASE=Entry(myWindow)
entryUSERNAME=Entry(myWindow)
entryPASSWORD=Entry(myWindow)
entryTableName=Entry(myWindow)
entryHOSTNAME.grid(row=2, column=1)
entryDATABASE.grid(row=3, column=1)
entryTableName.grid(row=4, column=1)
entryUSERNAME.grid(row=6, column=1)
entryPASSWORD.grid(row=7, column=1)


b3=Button(myWindow, text='Run: full execution',relief='ridge',width=45, height=2, command = run)
b3.grid(row=8, columnspan=2, sticky=W, padx=5, pady=5)

b4=Button(myWindow, text='Run: import csv to SQL only',relief='ridge',width=45, height=2, command = run_csv)
b4.grid(row=9, columnspan=2, sticky=W, padx=5, pady=5)


myWindow.mainloop()

Xevent Reader
http://example.com/2022/08/17/Xevent Reader/
Author
Jiacheng Xie
Posted on
August 17, 2022
Licensed under