(20260109) Almost done with Cosec-TCAOFF sync.

2026-01-09 16:35:49 +05:30
parent d41bf34101
commit 5bec648313
11 changed files with 1809 additions and 163 deletions
@@ -0,0 +1,141 @@
+# IPDR at VKNTPL
+
+Implementation by **Bhushan C Thakkar** and **Khushal P Soonderji** in November of 2025.
+
+_Documentation made on 20251128_
+
+---
+
+## 1. Login:
+
+The username is `vknipdr` and the server address is `ipdr.prysmnet.com`. You may SSH into the device to perform your activities.
+
+>ssh vknipdr@ipdr.prysmnet.com
+
+---
+
+## 2. Capturing Logs With Syslog-ng:
+
+The system uses **syslog-ng** to capture IPDR-related logs from **Mikrotik** hardware and saves them to a file on the server in fast and efficient **CSV format**.
+
+The file that hold the config can be accessed by the following command (`sudo` required):
+
+> nano /etc/syslog-ng/conf.d/mikrotik.conf
+
+Here is the code snippet that makes the magic work:
+
+```text
+# ------------------------------------------------------------
+# SOURCE — receive logs from MikroTik
+# ------------------------------------------------------------
+source s_mikrotik_nat {
+    udp(ip(0.0.0.0) port(514) keep-hostname(yes));
+
+};
+
+# ------------------------------------------------------------
+# Filter - accept on Forward Chains
+# ------------------------------------------------------------
+
+filter f_forward {
+    match("forward:") and not match("src-mac");
+};
+
+
+# ------------------------------------------------------------
+# PARSER — extract NAT fields using regex (flat declaration)
+# ------------------------------------------------------------
+
+parser p_nat_forward_regex {
+        regexp-parser(
+                template("${MESSAGE}")
+                patterns(
+                        "forward: in:(?<in>.*) out:(?<out>.*), proto (?<proto>.*), (?<src_ip>[\\d\\.]*):(?<src_port>[\\d]+)->(?<dst_ip>[\\d\\.]*):(?<dst_port>[\\d]+)(?:, NAT \\(.*?(?<nat_trans_ip>[\\d\\.]+):(?<nat_trans_port>[\\d]+)\\)[^,]*)?, len (?<len>\\d+)"
+                )
+        );
+};
+
+# ------------------------------------------------------------
+# TEMPLATE —
+# ------------------------------------------------------------
+template t_nat_csv {
+        template("${DATE},$src_ip,$src_port,$nat_trans_ip,$nat_trans_port,$dst_ip,$dst_port\n");
+        template-escape(no);
+};
+
+# ------------------------------------------------------------
+# DESTINATION — MongoDB
+# ------------------------------------------------------------
+
+
+# ------------------------------------------------------------
+# DESTINATION — Raw log file (for testing)
+# ------------------------------------------------------------
+destination d_rawfile {
+        file(
+                "/mnt/storage/syslog-ng/prysmnet/mikrotik-raw.log"
+                template(t_nat_csv)
+                flush_lines(1)
+        );
+};
+
+# ------------------------------------------------------------
+# LOG PATH — Write all incoming MikroTik messages directly to file
+# ------------------------------------------------------------
+log {
+    source(s_mikrotik_nat);
+    filter(f_forward);
+    parser(p_nat_forward_regex);
+    destination(d_rawfile);
+};
+```
+
+---
+
+## 3. Restarting Syslog-ng:
+
+Once you have saved your config file, you will need to restart the process with the following command (`sudo` required):
+
+> systemctl restart syslog-ng
+
+---
+
+## 4. Checking The Logs:
+
+You can see a *live trail* of the logs by running the following command:
+
+> tail -f -n 10 /mnt/storage/syslog-ng/prysmnet/mikrotik-raw.log
+
+You will see the logs in the following format:
+
+```text
+Nov 11 14:39:03,10.252.247.94,54084,103.171.2.187,54084,142.250.194.234,443
+Nov 11 14:39:03,10.252.255.135,56424,103.171.2.219,56424,123.63.54.23,443
+Nov 11 14:39:03,10.252.247.94,54084,,,142.250.194.234,443
+```
+
+---
+
+## 5. Log Rotation:
+
+Logs can add up very fast and the file can become unreasonable large and tough to manage. For this, we need to perform
+log rotation. A service named `logrotate` has been used for the same. At the time of first setup, the file was set up such
+that the configuration could be seen by running the following command:
+
+> cat /etc/logrotate.d/syslog-forward
+
+And the contents were:
+
+```text
+/var/log/syslog-ng/mikrotik-raw.log{
+   size 20G
+   rotate 365
+   compress
+   delaycompress
+   missingok
+   notifempty
+   copytruncate
+   dateext
+   dateformat -%Y-%m-%d_%H-%M-%S
+}
+```