Skip to content

Use escape character over double quoting in generated CSV #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Use escape character over double quoting in generated CSV #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
daac31e
favor escaping over double quoting example values
ebeahan Sep 21, 2020
7d7a774
update csv artifact
ebeahan Sep 21, 2020
9814477
changelog
ebeahan Sep 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Thanks, you're awesome :-) -->
#### Improvements

* Field details Jinja2 template components have been consolidated into one template #897
* Use backslashes as the escape characters over double double-quotes in CSV artifact. #972

#### Deprecated

Expand Down
42 changes: 21 additions & 21 deletions generated/csv/fields.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,base,@timestamp,date,core,,2016-05-23T08:05:34.853Z,Date/time when the event originated.
2.0.0-dev,true,base,labels,object,core,,"{""application"": ""foo-bar"", ""env"": ""production""}",Custom key/value pairs.
2.0.0-dev,true,base,labels,object,core,,"{\"application\": \"foo-bar\", \"env\": \"production\"}",Custom key/value pairs.
2.0.0-dev,true,base,message,text,core,,Hello World,Log message optimized for viewing in a log viewer.
2.0.0-dev,true,base,tags,keyword,core,array,"[""production"", ""env2""]",List of keywords used to tag each event.
2.0.0-dev,true,base,tags,keyword,core,array,"[\"production\", \"env2\"]",List of keywords used to tag each event.
2.0.0-dev,true,agent,agent.build.original,keyword,core,,"metricbeat version 7.6.0 (amd64), libbeat 7.6.0 [6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c built 2020-02-05 23:10:10 +0000 UTC]",Extended build information for the agent.
2.0.0-dev,true,agent,agent.ephemeral_id,keyword,extended,,8a4f500f,Ephemeral identifier of this agent.
2.0.0-dev,true,agent,agent.id,keyword,core,,8a4f500d,Unique identifier of this agent.
Expand All @@ -19,7 +19,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,client,client.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,client,client.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,client,client.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,client,client.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,client,client.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,client,client.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,client,client.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,client,client.geo.region_name,keyword,core,,Quebec,Region name.
Expand All @@ -42,7 +42,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,client,client.user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,client,client.user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,client,client.user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,client,client.user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,client,client.user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,cloud,cloud.account.id,keyword,extended,,666777888999,The cloud account or organization id.
2.0.0-dev,true,cloud,cloud.account.name,keyword,extended,,elastic-dev,The cloud account name.
2.0.0-dev,true,cloud,cloud.availability_zone,keyword,extended,,us-east-1c,Availability zone in which this host is running.
Expand All @@ -69,7 +69,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,destination,destination.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,destination,destination.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,destination,destination.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,destination,destination.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,destination,destination.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,destination,destination.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,destination,destination.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,destination,destination.geo.region_name,keyword,core,,Quebec,Region name.
Expand All @@ -92,7 +92,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,destination,destination.user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,destination,destination.user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,destination,destination.user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,destination,destination.user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,destination,destination.user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,dll,dll.code_signature.exists,boolean,core,,true,Boolean to capture if a signature is present.
2.0.0-dev,true,dll,dll.code_signature.status,keyword,extended,,ERROR_UNTRUSTED_ROOT,Additional information about the certificate status.
2.0.0-dev,true,dll,dll.code_signature.subject_name,keyword,core,,Microsoft Corporation,Subject name of the code signer
Expand All @@ -103,7 +103,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,dll,dll.hash.sha256,keyword,extended,,,SHA256 hash.
2.0.0-dev,true,dll,dll.hash.sha512,keyword,extended,,,SHA512 hash.
2.0.0-dev,true,dll,dll.name,keyword,core,,kernel32.dll,Name of the library.
2.0.0-dev,true,dll,dll.path,keyword,extended,,C:\Windows\System32\kernel32.dll,Full file path of the library.
2.0.0-dev,true,dll,dll.path,keyword,extended,,"C:\Windows\System32\kernel32.dll",Full file path of the library.
2.0.0-dev,true,dll,dll.pe.architecture,keyword,extended,,x64,CPU architecture target for the file.
2.0.0-dev,true,dll,dll.pe.company,keyword,extended,,Microsoft Corporation,"Internal company name of the file, provided at compile-time."
2.0.0-dev,true,dll,dll.pe.description,keyword,extended,,Paint,"Internal description of the file, provided at compile-time."
Expand Down Expand Up @@ -162,7 +162,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,event,event.type,keyword,core,array,,Event type. The third categorization field in the hierarchy.
2.0.0-dev,true,event,event.url,keyword,extended,,https://mysystem.example.com/alert/5271dedb-f5b0-4218-87f0-4ac4870a38fe,Event investigation URL
2.0.0-dev,true,file,file.accessed,date,extended,,,Last time the file was accessed.
2.0.0-dev,true,file,file.attributes,keyword,extended,array,"[""readonly"", ""system""]",Array of file attributes.
2.0.0-dev,true,file,file.attributes,keyword,extended,array,"[\"readonly\", \"system\"]",Array of file attributes.
2.0.0-dev,true,file,file.code_signature.exists,boolean,core,,true,Boolean to capture if a signature is present.
2.0.0-dev,true,file,file.code_signature.status,keyword,extended,,ERROR_UNTRUSTED_ROOT,Additional information about the certificate status.
2.0.0-dev,true,file,file.code_signature.subject_name,keyword,core,,Microsoft Corporation,Subject name of the code signer
Expand Down Expand Up @@ -233,7 +233,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,host,host.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,host,host.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,host,host.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,host,host.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,host,host.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,host,host.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,host,host.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,host,host.geo.region_name,keyword,core,,Quebec,Region name.
Expand Down Expand Up @@ -263,7 +263,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,host,host.user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,host,host.user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,host,host.user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,host,host.user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,host,host.user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,http,http.request.body.bytes,long,extended,,887,Size in bytes of the request body.
2.0.0-dev,true,http,http.request.body.content,keyword,extended,,Hello world,The full HTTP request body.
2.0.0-dev,true,http,http.request.body.content.text,text,extended,,Hello world,The full HTTP request body.
Expand Down Expand Up @@ -318,7 +318,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,observer,observer.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,observer,observer.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,observer,observer.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,observer,observer.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,observer,observer.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,observer,observer.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,observer,observer.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,observer,observer.geo.region_name,keyword,core,,Quebec,Region name.
Expand Down Expand Up @@ -437,11 +437,11 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,process,process.working_directory,keyword,extended,,/home/alice,The working directory of the process.
2.0.0-dev,true,process,process.working_directory.text,text,extended,,/home/alice,The working directory of the process.
2.0.0-dev,true,registry,registry.data.bytes,keyword,extended,,ZQBuAC0AVQBTAAAAZQBuAAAAAAA=,Original bytes written with base64 encoding.
2.0.0-dev,true,registry,registry.data.strings,keyword,core,array,"[""C:\rta\red_ttp\bin\myapp.exe""]",List of strings representing what was written to the registry.
2.0.0-dev,true,registry,registry.data.strings,keyword,core,array,"[\"C:\rta\red_ttp\bin\myapp.exe\"]",List of strings representing what was written to the registry.
2.0.0-dev,true,registry,registry.data.type,keyword,core,,REG_SZ,Standard registry type for encoding contents
2.0.0-dev,true,registry,registry.hive,keyword,core,,HKLM,Abbreviated name for the hive.
2.0.0-dev,true,registry,registry.key,keyword,core,,SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winword.exe,Hive-relative path of keys.
2.0.0-dev,true,registry,registry.path,keyword,core,,HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winword.exe\Debugger,"Full path, including hive, key and value"
2.0.0-dev,true,registry,registry.key,keyword,core,,"SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winword.exe",Hive-relative path of keys.
2.0.0-dev,true,registry,registry.path,keyword,core,,"HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winword.exe\Debugger","Full path, including hive, key and value"
2.0.0-dev,true,registry,registry.value,keyword,core,,Debugger,Name of the value written.
2.0.0-dev,true,related,related.hash,keyword,extended,array,,All the hashes seen on your event.
2.0.0-dev,true,related,related.hosts,keyword,extended,array,,All the host identifiers seen on your event.
Expand All @@ -467,7 +467,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,server,server.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,server,server.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,server,server.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,server,server.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,server,server.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,server,server.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,server,server.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,server,server.geo.region_name,keyword,core,,Quebec,Region name.
Expand All @@ -490,7 +490,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,server,server.user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,server,server.user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,server,server.user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,server,server.user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,server,server.user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,service,service.ephemeral_id,keyword,extended,,8a4f500f,Ephemeral identifier of this service.
2.0.0-dev,true,service,service.id,keyword,core,,d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6,Unique identifier of the running service.
2.0.0-dev,true,service,service.name,keyword,core,,elasticsearch-metrics,Name of the service.
Expand All @@ -508,7 +508,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,source,source.geo.continent_name,keyword,core,,North America,Name of the continent.
2.0.0-dev,true,source,source.geo.country_iso_code,keyword,core,,CA,Country ISO code.
2.0.0-dev,true,source,source.geo.country_name,keyword,core,,Canada,Country name.
2.0.0-dev,true,source,source.geo.location,geo_point,core,,"{ ""lon"": -73.614830, ""lat"": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,source,source.geo.location,geo_point,core,,"{ \"lon\": -73.614830, \"lat\": 45.505918 }",Longitude and latitude.
2.0.0-dev,true,source,source.geo.name,keyword,extended,,boston-dc,User-defined description of a location.
2.0.0-dev,true,source,source.geo.region_iso_code,keyword,core,,CA-QC,Region ISO code.
2.0.0-dev,true,source,source.geo.region_name,keyword,core,,Quebec,Region name.
Expand All @@ -531,7 +531,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,source,source.user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,source,source.user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,source,source.user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,source,source.user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,source,source.user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,span,span.id,keyword,extended,,3ff9a8981b7ccd5a,Unique identifier of the span within the scope of its trace.
2.0.0-dev,true,threat,threat.framework,keyword,extended,,MITRE ATT&CK,Threat classification framework.
2.0.0-dev,true,threat,threat.tactic.id,keyword,extended,array,TA0002,Threat tactic id.
Expand Down Expand Up @@ -632,7 +632,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,url,url.original,keyword,extended,,https://www.elastic.co:443/search?q=elasticsearch#top or /search?q=elasticsearch,Unmodified original url as seen in the event source.
2.0.0-dev,true,url,url.original.text,text,extended,,https://www.elastic.co:443/search?q=elasticsearch#top or /search?q=elasticsearch,Unmodified original url as seen in the event source.
2.0.0-dev,true,url,url.password,keyword,extended,,,Password of the request.
2.0.0-dev,true,url,url.path,keyword,extended,,,"Path of the request, such as ""/search""."
2.0.0-dev,true,url,url.path,keyword,extended,,,"Path of the request, such as \"/search\"."
2.0.0-dev,true,url,url.port,long,extended,,443,"Port of the request, such as 443."
2.0.0-dev,true,url,url.query,keyword,extended,,,Query string of the request.
2.0.0-dev,true,url,url.registered_domain,keyword,extended,,example.com,"The highest registered url domain, stripped of the subdomain."
Expand All @@ -650,7 +650,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,user,user.id,keyword,core,,,Unique identifier of the user.
2.0.0-dev,true,user,user.name,keyword,core,,albert,Short name or login of the user.
2.0.0-dev,true,user,user.name.text,text,core,,albert,Short name or login of the user.
2.0.0-dev,true,user,user.roles,keyword,extended,array,"[""kibana_admin"", ""reporting_user""]",Array of user roles at the time of the event.
2.0.0-dev,true,user,user.roles,keyword,extended,array,"[\"kibana_admin\", \"reporting_user\"]",Array of user roles at the time of the event.
2.0.0-dev,true,user_agent,user_agent.device.name,keyword,extended,,iPhone,Name of the device.
2.0.0-dev,true,user_agent,user_agent.name,keyword,extended,,Safari,Name of the user agent.
2.0.0-dev,true,user_agent,user_agent.original,keyword,extended,,"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1",Unparsed user_agent string.
Expand All @@ -664,7 +664,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,user_agent,user_agent.os.platform,keyword,extended,,darwin,"Operating system platform (such centos, ubuntu, windows)."
2.0.0-dev,true,user_agent,user_agent.os.version,keyword,extended,,10.14.1,Operating system version as a raw string.
2.0.0-dev,true,user_agent,user_agent.version,keyword,extended,,12.0,Version of the user agent.
2.0.0-dev,true,vulnerability,vulnerability.category,keyword,extended,array,"[""Firewall""]",Category of a vulnerability.
2.0.0-dev,true,vulnerability,vulnerability.category,keyword,extended,array,[\"Firewall\"],Category of a vulnerability.
2.0.0-dev,true,vulnerability,vulnerability.classification,keyword,extended,,CVSS,Classification of the vulnerability.
2.0.0-dev,true,vulnerability,vulnerability.description,keyword,extended,,"In macOS before 2.12.6, there is a vulnerability in the RPC...",Description of the vulnerability.
2.0.0-dev,true,vulnerability,vulnerability.description.text,text,extended,,"In macOS before 2.12.6, there is a vulnerability in the RPC...",Description of the vulnerability.
Expand Down
2 changes: 2 additions & 0 deletions scripts/generators/csv_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ def save_csv(file, sorted_fields, version):
with open(file, open_mode) as csvfile:
schema_writer = csv.writer(csvfile,
delimiter=',',
doublequote=False,
escapechar='\\',
quoting=csv.QUOTE_MINIMAL,
lineterminator='\n')

Expand Down