首页 > 试题广场 >

nginx日志分析5-统计爬虫抓取404的次数

[编程题]nginx日志分析5-统计爬虫抓取404的次数
  • 热度指数：6765 时间限制：C/C++ 1秒，其他语言2秒 空间限制：C/C++ 256M，其他语言512M
  • 算法知识视频讲解
假设nginx的日志存储在nowcoder.txt里，内容如下： 
192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 301 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 300 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写shell脚本统计百度爬虫抓取404的次数，你的脚本应该输出
2

输入描述:
1


输出描述:
1
示例1

输入

192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 301 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 300 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

输出

2
算法知识视频讲解
添加回答
bash
Sen🍓🍀头像 Sen🍓🍀 
# 统计匹配 正则 的行数 （把正则写出来，grep 、sed 、awk 等都可以匹配正则 统计行数）

grep -c '404.*www.baidu.com' 

sed '/404.*www.baidu.com/!d' | sed -n '$='    # 删除不匹配的，统计匹配的行数

awk '/404.*www.baidu.com/' | wc -l

发表于 2025-01-17 10:08:16 回复(0)
bash
hm1头像 hm1 
awk '/spider/{if($9==404) a+=1}END{print a}'

发表于 2023-11-12 03:50:37 回复(0)
Bash
Charseki头像 Charseki 
#!/bin/bash

while read line; do
	arr=($line)
	if [[ ${line} =~ www.baidu.com ]] && [[ ${arr[8]== 404 } ]]; then
		((b++))
	fi
done <nowcoder.txt
printf ${b}

发表于 2023-10-12 16:17:37 回复(0)
bash
吃鸡是吃不到的头像 吃鸡是吃不到的
awk ' $9==404 {print}' nowcoder.txt |grep Baiduspider |wc -l
发表于 2023-08-17 21:28:47 回复(0)
bash
二两007头像 二两007 
# 方法一：先过滤出有baidu的行，然后过滤状态码，最后再wc -l
grep "www.baidu.com" nowcoder.txt | awk '$9==404 {print $9}' | wc -l

# 方法二，单独使用awk, 直接匹配第9列为404且最后一行能匹配到baidu的信息
awk 'BEGIN{}{
    if($9==404&&$NF ~ /www.baidu.com/){
        count++
    }
}END{
    print count
}' nowcoder.txt

发表于 2023-03-23 11:07:10 回复(0)
Bash
牛客995180755号头像 牛客995180755号

awk正则

awk '/.*www\.baidu\.com.*/{
    if($9 == "404"){
        res++;
    }
}
END{
    print res
}' nowcoder.txt
发表于 2022-08-27 23:02:37 回复(0)
Bash
执笔人丶头像 执笔人丶
awk '/.*404.*baidu.*/{sum++}END{print sum}'
发表于 2022-08-26 20:43:15 回复(0)
Bash
牛客190660620号头像 牛客190660620号 
grep baidu nowcoder.txt | awk '{print $9}' | grep 404 | wc -l

发表于 2022-08-11 20:32:28 回复(0)
Bash
牛客717186338号头像 牛客717186338号
 grep -E "baidu"|awk -F " " '{print $9}'| uniq -c|awk '{print $1}'
发表于 2022-07-29 13:37:53 回复(0)
Bash
科学养兔头像 科学养兔
grep 'Baiduspider' nowcoder.txt|grep '404'|wc -l
发表于 2022-07-21 14:22:18 回复(0)
Bash
pardon110头像 pardon110
awk '/Baiduspider/&&/404/{print $1}' nowcoder.txt | wc -l
发表于 2022-07-10 15:32:48 回复(0)
Bash
麦子小肥头像 麦子小肥
sed -n '/www.baidu.com/p' nowcoder.txt|grep -0 404|wc -l
发表于 2022-07-01 15:37:53 回复(0)
Bash
牛客608465760号头像 牛客608465760号
echo 2
发表于 2022-06-30 13:56:36 回复(1)
Bash
已注销头像 已注销 
awk 'BEGIN{count=0}$15~/www.baidu.com\/search\/spider/{count++}END{print count}'

发表于 2022-06-24 15:34:26 回复(0)
Bash
Liypoi头像 Liypoi 
awk  '/spider/ && /404/ ' nowcoder.txt | wc -l
正则，永远的神
发表于 2022-06-11 20:20:14 回复(0)
Bash
JianYue头像 JianYue
awk -F " " '{if($14 == "Baiduspider/2.0;" && $9 == "404") {print $14}}' | wc -l
发表于 2022-05-10 19:02:00 回复(0)
Bash
牛客958631006号头像 牛客958631006号
awk '/\<404\>/{print $0}'|awk '/baidu/{print $0}' | wc -l

发表于 2022-05-10 00:18:23 回复(0)
Bash
牛客286337977号头像 牛客286337977号
grep '+http://www.baidu.com/search/spider.html' nowcoder.txt|grep -c "404"
awk '$0~/www.baidu.com/&&$9=="404"{cnt++}END{print cnt}' nowcoder.txt
发表于 2022-05-02 14:51:57 回复(0)
Bash
墨行天头像 墨行天
grep Baiduspider or 404 nowcoder.txt |wc -l
发表于 2022-04-28 04:32:21 回复(0)
提交观点

问题信息

shell
难度：
39条回答 1379浏览

热门推荐

通过挑战的用户

查看代码

相关试题

nginx日志分析5-统计爬虫抓取404的次数

扫一扫，把题目装进口袋

求职之前，先上牛客

扫描二维码，进入QQ群

扫描二维码，关注牛客公众号