Recently some developers have been asking about vulnerability management (VM) custom fields attached to vulnerabilities. In attempting to answer their questions, it was discovered the API documentation was incorrect. Now the documentation has been fixed, I thought a blog on custom fields would be helpful.
If you're curious about why this is Part Two, it is because both Part One and Part Two are about metadata. Part One is about Asset metadata while Part Two is about vulnerability metadata.
Custom fields are additional metadata to vulnerabilities. They add context. For example, when creating a CISA risk meter, the CISA custom field was created to indicate the vulnerability was on the CISA list. This blog will discuss how to search for and update custom fields with APIs. It will also discuss code that lists unique custom fields and their values.
Custom Field Search
To search for a custom field, the "Search Vulnerabilities" API is used. The query parameter custom_fields is used in the following manner:
custom_fields:<custom_field>[]=<search_value>
For example:
https://api.kennasecurity.com/vulnerabilities/search?custom_fields:Category[]=good
If you want to search for more than one custom field, I recommend:
https://api.kennasecurity.com/vulnerabilities/search?custom_fields:Category[]=good&custom_fields:CISA[]=true
Note that custom_fields: with the custom field and [] after the custom field is required. Unfortunately, wildcarding is not allowed. Some developers might think that this query parameter is poorly defined. I would agree, and we know about it, but we're stuck with it for now.
Here are two code examples:
In build_cisa_risk_meter.py:
80 # Assemble the search URL.
81 page_size_query = f"per_page={SEARCH_PAGE_SIZE}"
82 custom_field_query = f"custom_fields:{custom_field}[]=none"
83 search_url = f"{base_url}vulnerabilities/search?{page_size_query}&{custom_field_query}"
And in search_custom_fields.py:
38 page_size_query = f"per_page={page_size}"
39 custom_field_query = f"custom_fields:{search_custom_field}[]={search_value}"
40 search_url = f"{base_url}vulnerabilities/search?{page_size_query}&{custom_field_query}"
The script search_custom_fields.py is provided in the custom_fields directory of Kenna's blog_samples repo as a simple custom field search script. It takes two command line parameters, custom_field name, and custom_field value, and returns the basic information on the vulnerabilities that contain the desired custom field and the custom field value. Feel free to enhance.
Custom Field Update
Let's say you have a custom field, risk_accepted_expiration_date. You want a user interface or a small script to update the risk accepted expiration date. To do this you will need to use the "Update Vulnerabilities" API. It can be used to update custom fields of one vulnerability. Here is an example below from update_custom_field.py:
16 def update_vuln(base_url, headers, vuln_id, custom_field_id, custom_field_value):
17 update_url = f"{base_url}vulnerabilities/{vuln_id}"
18 update_custom_field_id = f"{custom_field_id}"
19 update_data = {
20 "vulnerability": {
21 "custom_fields": {
22 update_custom_field_id: custom_field_value
23 }
24 }
25 }
26
27 print(f"Update URL: {update_url}")
28 print_json(update_data)
29
30 # Invoke the update vulnerability endpoint.
31 response = requests.put(update_url, headers=headers, data=json.dumps(update_data))
32 if response.status_code != 204:
33 print("Vulnerability Update API ", response, update_url)
34 sys.exit(1)
Since this is an update, the custom field information is in the HTTP request body. To update a custom field, the custom field ID is required along with the new value to update with. To obtain the custom field ID, invoke the List or Search Vulnerabilities APIs.
Custom Field Bulk Update
To update a custom field for a list of vulnerabilities, the "Bulk Update Vulnerabilities" API is used. This was discussed in the "How to Create a CISA Risk Meter" blog, section "Bulk Vulnerabilities Update," but I'm going to review it here. The HTTP request body is the same when it comes to custom fields; however, now vulnerability_ids to be updated are included in the HTTP request body. Here is an example from build_cisa_risk_meter.py:
152 def update_cisa_vulns(base_url, headers, vuln_ids, custom_field_id):
153 if len(vuln_ids) == 0:
154 return
155
156 bulk_update_url = f"{base_url}vulnerabilities/bulk"
157 update_custom_field_id = f"{custom_field_id}"
158 update_data = {
159 "vulnerability_ids": vuln_ids,
160 "vulnerability": {
161 "custom_fields": {
162 update_custom_field_id: "true"
163 }
164 }
165 }
166
167 response = requests.put(bulk_update_url, headers=headers, data=json.dumps(update_data))
Again, the custom field ID along with the new custom field value is required.
List Unique Custom Fields
Saving the largest topic for last. One issue with using custom fields is there is no way to obtain a list of custom fields being used for all your vulnerabilities. Are they still valid? What values are you using? I wrote the script, blog_list_custom_fields.py, to create a list of custom fields with a usage count, and custom field values, and written to a CSV file.
Data Structures
Let's look at the script's data structures. There is a class, Custom_Field that contains the custom field name, ID, unique values, and count. It contains methods to add new custom field values, return the values, and return the number of values. Looking at it, you will notice that the code checks if the value is None.
29 # Append a custom field value to the list of values.
30 def append_new_value(self, custom_field_value):
31 if custom_field_value is None:
32 return
33
34 self.custom_field_count += 1
35
36 # If the value is not in the list of values, add it.
37 if not custom_field_value in self.custom_field_values:
38 self.custom_field_values.append(custom_field_value)The reason is once a custom field is defined, it is attached to all vulnerabilities with a null value. This script is only interested in non-null values (None in python). If the value is not, it is set for this vulnerability and therefore placed in the values array provided that it is not already in the array.
The other data structure is the dictionary, unique_custom_fields. The custom field name is the key and it maps to an Custom_Field object. It is passed by reference to the appropriate functions.
331 # A dictionary of custom fields keyed by custom field name with the value
332 # of a custom field object.
333 unique_custom_fields = {}
Obtaining Vulnerability Information
First, the script does a vulnerability export. This is very similar to an asset export. This was covered in the blog "Unique Asset Tags - Part 1". The only difference is that model in export_settings is set to vulnerability instead of asset for the "Request Data Export" API.
filter_params = {
'status' : ['open'],
'export_settings': {
'format': 'jsonl',
'model': 'vulnerability'
}
}
Please read "Unique Asset Tags - Part 1" to understand Data Exports. The vulnerability data export code produces a file vuln<search_id>.jsonl. which contains all the vulnerabilities in a JSONL format.
process_vuln_export()
Reads the JSONL file line by line converting each line to a JSON dictionary. Each line is a vulnerability. If there are custom_fields in the vulnerability, then further processing is required in process_custom_fields(). A dot is produced for every 1000 vulnerabilities processed. Hopefully, this status won't keep you hanging and indicates processing is begin performed.
241 # Process vulnerabilities in the JSONL format.
242 def process_vuln_export(jsonl_vuln_file_name, unique_custom_fields):
243 print_info(f"Opening {jsonl_vuln_file_name} for processing.")
244 logging_interval = 1000
245
246 # Open the JSONL file and read it line by line, checking each vulnerability line for custom fields.
247 with open(jsonl_vuln_file_name, 'r') as jsonl_f:
248 for line_num, vuln_line in enumerate(jsonl_f):
249 vuln = convert_to_json(vuln_line)
250 if "custom_fields" in vuln:
251 logging.info(f"Found custom_field in Vuln {line_num}")
252 process_custom_fields(unique_custom_fields, vuln["custom_fields"])
253
254 if (line_num + 1) % logging_interval == 0:
255 print(".", end='', flush='True')
256
257 print("")
258 return (line_num + 1)
process_custom_fields()
This function decides in the custom field is unique. It does this by checking if the custom field name key is in unique_custom_fields (line 232).
224 # Process an array of custom fields in a vulnerability.
225 def process_custom_fields(unique_custom_fields, custom_fields):
226
227 # Process one custom field at a time.
228 for custom_field in custom_fields:
229 cf_name = custom_field["name"]
230
231 logging.debug(f"Processing custom field: {cf_name}")
232 if cf_name in unique_custom_fields:
233 unique_custom_field = unique_custom_fields[cf_name]
234 if unique_custom_field.custom_field_id != custom_field["custom_field_definition_id"]:
235 print_error(f"IDs for custom field {cf_name} do not match. " +
236 f"{unique_custom_field.custom_field_id}, {custom_field['custom_field_definition_id']}")
237 unique_custom_field.append_new_value(custom_field["value"])
238 else:
239 append_custom_field(unique_custom_fields, custom_field)
240If the custom field is in the dictionary, then the value is added to the custom field by calling append_new_value() in line 237, else a new custom field entry is created by calling append_custom_field() in line 239.
append_custom_field()
This is the function where a new Custom_Field object is created and appended to the unique_custom_fields dictionary.
218 # Append a custom field to the dictionary of unique custom fields.
219 def append_custom_field(unique_custom_fields, custom_field):
220 cf_name = custom_field["name"]
221 a_custom_field = Custom_Field(custom_field)
222 unique_custom_fields[cf_name] = a_custom_field
As a reminder, the custom field name is used as a key.
write_csv_file()
Finally, we get to the function that writes the unique custom field information to the CSV file, uniq_custom_fields.csv.
260 # Write the information out to a CSV file.
261 def write_csv_file(custom_fields):
262 # Open the CSV file and write the header row.
263 csv_file_name = "uniq_custom_fields.csv"
264 uniq_custom_fields_fp = open(csv_file_name, 'w', newline='')
265 uniq_custom_field_writer = csv.writer(uniq_custom_fields_fp)
266 uniq_custom_field_writer.writerow(["Custom Field", "Custom Field ID", "Field Count", "Value Count", "Custom Field Values"])
267
268 # Process each custom field and write it to the CSV file.
269 for custom_field_key in custom_fields:
270 custom_field = custom_fields[custom_field_key]
271
272 if custom_field.custom_field_count == 0:
273 uniq_custom_field_writer.writerow([custom_field.custom_field_name, custom_field.custom_field_id, custom_field.custom_field_count])
274 else:
275 uniq_custom_field_writer.writerow([custom_field.custom_field_name, custom_field.custom_field_id,
276 custom_field.custom_field_count, str(custom_field.get_num_values()), custom_field.get_values()])
277
278 uniq_custom_fields_fp.close()
279 print_info(f"{csv_file_name} is now available.")
There are two types of rows, one for null custom field values and one for non-null custom field values.
- null value row contains: custom field name, ID, and a zero count. (line 273)
- a non-null row contains: custom field name, ID, non-zero count, number of unique values, and the unique values. (lines 275-276)
If the count is zero, that implies that no one is currently using the custom field. It could be time to remove the unused custom field.
CSV Output
Here is an example of the CSV output:
Above is an Excel representation of the unique_custom_fields.csv. As you can see the spreadsheet has "Custom Field," "Custom Field ID," "Field Count," "Value Count," and "Custom Field Values." If the "Field Count" is zero, the custom field is currently not being used. "Custom Field Values is a comma separate string; for example, "Bad, good, extract, excellent" are contained in one cell.
Conclusion
After reading this blog, my hope is as a developer, you will know how to work with VM custom fields and the code examples will inspire you to write your own scripts. The new code mentioned in this blog is located in Kenna Security's Github blog_samples repository under the python/custom_fields directory.
Until next time,
API Evangelist
This blog was originally written for Kenna Security, which has been acquired by Cisco Systems.
Learn more about Cisco Vulnerability Management.