I have found that in some cases, running log analysis tools can be complicated to learn and cumbersome to support, especially for smaller-scale projects. Each project or company needs the log analysis capability: some large commercial tools and software are excessive or overkill.

Here at 2T Security we are hosting a web service which offers one of our UK clients visualisations of some large datasets in a read-only format, which is for the consumption of external nationally based third parties. This web service is hosted on Amazon Web Service (AWS) with S3 buckets focused with Web Application Firewall (WAF) and application logging.

As this is an external web service, we have implemented security measures and the tool we are using for monitoring and analytics is SpectX.

SpectX is used as a log analyser for data; analytics, investigation and manipulation. It reads and parses raw log files to return a structured and accessible format. SpectX is directed to the data source of the web service and only returns the result of the script and not the raw data. SpectX does not require additional storage locally to query data as it pulls directly from the source for processing.

One example of how we monitor our web service is by identifying attempts to the webservice and by using open source threat feeds we have created a more broader security monitoring aspect:

SpectX is given the API credentials to gain access to the S3 buckets in a single pane window. Examples have been anonymised below.

Using SpectX Parsing rules, I can identify what the logs look like and what the output should be displayed as:

The pattern defined above is for an AWS WAF to highlight all blocked attempts to the web service for the last 10 days. By using this pattern, I can increase the extraction of data fields to return the IP address in question with the geo tag and country code.

Using SpectX  I can break apart the logs generated by AWS and display them in an easily readable format. The results are displayed as follows:

The results have produced a list of IPs attempting access with the geolocation and country code attached. This can be displayed with graphics such as maps:

Using this information, I can collate and compare the data to query other results such as known bad IP lists, in this case IPsum. Though there are many open source threat feeds available to use, IPsum is my preferred as it is updated daily with the latest known bad IPs collated from various sources then pushed to a grabbable github repository. There numerous others which can be interchangeable, such as Blocklist or CIA army.

Inputting data to SpectX is fairly straightforward. The URL used for the source is parsed and returned with a preview:

When preparing a query, it contains the prewritten pattern. In this instance, I need the IP address, therefore I will select only this. The below pattern will pull the most recent master list of IPs:

SpectX will simultaneously run the two queries (“IPsum” and “known bad lookups”) to pull all blocked attempts to the webservice, whilst querying the most recent version of IPsum’s data.

$pattern = <<<PATTERN
   JSON{
   TIMESTAMP('S'):timestamp,
   STRING:action,
   STRING:terminatingRuleId
   }(greedy='AWSX'):AWS EOL
PATTERN;
LIST('s3s://WEBSERVICE-waf/$yyyy$/$MM$/$dd$/*/aws-waf-logs-WEBSERVICE-1*')
| filter (path_time >= now()[-135 day] AND path_time <= now()[+1 day])
| parse(pattern:$pattern)
| select(AWS[timestamp], AWS[action],AWS[terminatingRuleId], IPADDR(AWS[AWSX][httpRequest][clientIp]) as IP)
| select (count(IP) as Hits, IP)
| group(IP)
| join(@[/PATH/Report/Bad IPSUM List.sx] on left.IP = right.IPaddress)
| select(*, geo(IP), CC(IP))
| sort(Hits DESC)
| select(*, case_when:
         CASE
           WHEN cc='--' THEN (DNS_LOOKUP(IP))
	   WHEN cc='' THEN (DNS_LOOKUP(IP))
           ELSE CC(IP)
         END)

As a result, comparing both the known bad IPs with the blocked IP attempts returns with the following:

With this result, I can clearly see who has been attempting to gain access to the webservice from an identified and known bad IP address. In this instance although the 195.54.160.135 IP address was not listed as a country, I also ran a DNS Lookup for further understanding into the IP owner: this returned null. After a domain lookup, I was able to identify that the address has been reported as abusive from a Russian domain.

I then ran this same query with successful login attempts to be able to check if there are any known bad IPs gaining access to the network. Fortunately, there are none.

Other instances in which SpectX might be used is to recognise any anomalies. For example, when trying to find the most popular HTTP version visiting the webservice the below was found:

$pattern = <<<PATTERN
   JSON{
   TIMESTAMP('S'):timestamp,
   STRING:action
   }(greedy='AWSX'):AWS EOL
PATTERN;
LIST('s3s://WEBSERVICE-waf/$yyyy$/$MM$/$dd$/*/aws-waf-logs-WEBSERVICE-1*')
| filter(path_time BETWEEN T('2020-07-16 00:00:00 +0200') AND now()[+1 day])
| parse(pattern:$pattern)
| select(AWS[AWSX][httpRequest][httpVersion] as httpVersion)
| select (*, count(httpVersion) as Hits)
| group (httpVersion)
| sort(Hits DESC) 

Although HTTP versions 1.0/ 1.1/ 2 are all expected, one anomaly was identified as likely spoofed HTTP header. Upon closer inspection I found it was a blocked attempt from Romanian-based IP known for IP scanning.

2T Security is a partner of SpectX. We use it internally and we are now using it for regular monitoring to baseline what we know to be normal. The more frequently we use SpectX, the easier and more efficient our processes will become when identifying anomalies, security alerts and other negative or unrecognised activities.

For more information about SpectX and how we have utilised the tool with our own systems and projects please contact us enquiries@2t-security.com.