We’ve recently added two new features to WebSpy Vantage (all flavours) to deal with a specific issue setting up automated importing tasks using date modifiers in file masks.

A common setup is to have a task that creates a storage each month, then a separate task that runs at the end of each day – say at 10pm, to imports new hits into the storage.  The issue here is that at the end of the month, hits between 10pm and 12pm will never be imported, as the next time the task runs it will be dealing with next month’s storage.

As long as your log files contain the date in the file name, you can use date modifiers in file masks to import the appropriate log files. Date Modifiers are used to create file masks such as ‘access_200902*.log’, or ‘access_20090203.log’ that will cull down the number of log files to import. This is much faster than using date filters (specified on the filters page).

The Issue:

The issue is that a log file location in a storage can only have one file mask, and this cannot (until now) not be modified by task actions.  So you cannot specify a new task on the first of each month to import yesterday’s log file into last month’s storage, if that storage already has a file mask to only import the current day’s log file.

The two solutions:

1. Save the literal file mask into the storage when the storage gets created.

This option allows you to specify a file mask such as  access_%[yyyyMM]* in the ‘Import logs into new storage’ task, but when each storage is created, the actual file mask that gets saved into the storage includes the literal month and year values such as access_200902*

This way you can specify an ‘Import new hits’ job, and even if that job runs in the next month,  ONLY log files from the desired month will be checked for new hits.

To access this option:

1. Go to the Tasks screen and edit your ‘Import logs into new storage’ action
2. Go to the Input Selection page
3. Select your log folder (not individual files) and click Edit
4. Check the ‘Save literal date into mask’ check box


Save literal date into file mask option

Downside of this option is that in my ‘monthly storage’ example, all 31 log files for the month will be checked for new hits.

2. Override a storage’s file mask using the Import New Hits task action.

The Select Storage dialog that appears when configuring an ‘Import new hits into existing storage’ action now has a new option to override a storage’s file mask when the task runs.



Override the file mask option

This allows you to configure a task that runs on the first day of each month to import only yesterday’s log file into last month’s storage (need to add last month’s storage using the Add button, and specify the appropriate date modifier to access it – e.g. “%[-1m, yyyyMM] My Monthly Storage”).

This will only check one file for new hits and will finish the import job faster than option 1 (using the monthly storage example).

See also:

• Vantage Update 2.2.0.90
• How to Report on Enterprise Storages – Vantage Guidelines
• Increase Report Speed : Reduce What You Import
• Report Templates: Making The Most of Your Data
• Web Module – Dynamic Employee Internet Reports