今天在rk3568开发板上跑程序的时候发现mysql数据库的数据没读取出来,检查后发现是mysql没有运行,尝试启动mysql服务提示启动失败:job for mysql.service failed...

排查过程

查看mysql错误日志:cat /var/log/mysql/error.log

提示:Error number 28 means 'No space left on device'

查找相关资料后觉得有可能是磁盘空间爆满导致mysql启动失败,于是df -h查看磁盘使用情况,发现挂载在根目录下的磁盘确实是爆满了

疑问:为什么挂载在根目录下的磁盘爆满会导致mysql启动失败?
MySQL 需要写入数据文件、日志、临时文件(如 tmpdir)等。如果根目录(特别是 MySQL 的数据目录,通常是 /var/lib/mysql)所在的分区没有足够空间,写入操作会失败,进而导致服务无法启动。

网上的方法大多是清理根目录下没用的资源来释放内存。通过du -sh /* 发现根目录满的原因主要是因为太多的第三方库so文件在/usr/lib目录下了,想清理无从下手,不知道哪些是没用的资源。

首先理清一下磁盘分区和文件系统的关系

磁盘分区是将物理磁盘划分为多个逻辑部分,而文件系统是在每个分区上创建的逻辑结构,用于组织和管理文件和目录。磁盘分区提供了逻辑隔离和独立管理的功能,而文件系统则负责管理文件的存储和访问。

挂载实际上挂的是文件系统

https://blog.csdn.net/m0_65690223/article/details/131408110

所以上面的df -h显示的问题准确点来说是文件系统可使用空间不足导致的mysql启动失败

解决思路:
(1)能否给挂载在根目录下的文件系统扩容?
(2)根目录下占用内存较大的主要是/usr目录,能否把空闲的磁盘挂载在/usr/lib下?
(3)重新烧录镜像,对比根目录下哪些东西是后来添加的,把没用的清理

这里打算尝试方法一:对文件系统/dev/root进行扩容(此方法到后面会有问题,谨慎使用)

1、通过fdisk -l命令查看是否还有可用的磁盘空间,如果有可以磁盘空间可参照以下对文件系统进行扩容

linux对根目录扩容(命令行) - EastWood001 - 博客园

2、因为我的是开发板不是虚拟机,并且通过fdisk -l后发现没有多余磁盘空间,要想扩容磁盘只能更换更大的emmc/sd卡/硬盘,这显然是不现实的,因此需要先缩容再扩容
(对文件系统/dev/mmcblk0p8先缩容,再对/dev/root扩容)

(1)查看磁盘分区情况

root@TQ3568:fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 14.59 GiB, 15655239680 bytes, 30576640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D9700000-0000-4F75-8000-3B6C00005773

Device            Start      End  Sectors  Size Type
/dev/mmcblk0p1    16384    24575     8192    4M unknown
/dev/mmcblk0p2    24576    32767     8192    4M unknown
/dev/mmcblk0p3    32768   163839   131072   64M unknown
/dev/mmcblk0p4   163840   425983   262144  128M unknown
/dev/mmcblk0p5   425984   491519    65536   32M unknown
/dev/mmcblk0p6   491520 13074431 12582912    6G unknown
/dev/mmcblk0p7 13074432 13336575   262144  128M unknown
/dev/mmcblk0p8 13336576 30576575 17240000  8.2G unknown ##注意起始扇区13336576,后面会用到

(2)缩小/dev/mmcblk0p8 分区上的文件系统(/userdata)

root@TQ3568:umount /dev/mmcblk0p8  ##卸载分区
root@TQ3568:
root@TQ3568:
root@TQ3568:e2fsck -f /dev/mmcblk0p8  ##强制检查并修复 ext 系列文件系统
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
userdata: 16/1145664 files (6.3% non-contiguous), 290436/8620000 blocks
root@TQ3568:resize2fs /dev/mmcblk0p8 6144M   ##改/dev/mmcblk0p8分区文件系统的大小为6G
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/mmcblk0p8 to 6291456 (1k) blocks.
The filesystem on /dev/mmcblk0p8 is now 6291456 (1k) blocks long.

root@TQ3568:
root@TQ3568:lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0      179:0    0 14.6G  0 disk
├─mmcblk0p1  179:1    0    4M  0 part
├─mmcblk0p2  179:2    0    4M  0 part
├─mmcblk0p3  179:3    0   64M  0 part
├─mmcblk0p4  179:4    0  128M  0 part
├─mmcblk0p5  179:5    0   32M  0 part
├─mmcblk0p6  179:6    0    6G  0 part /
├─mmcblk0p7  179:7    0  128M  0 part /oem
└─mmcblk0p8  179:8    0  8.2G  0 part        ##分区大小没变,但文件系统缩小为6G
mmcblk0boot0 179:32   0    4M  1 disk
mmcblk0boot1 179:64   0    4M  1 disk
mmcblk3      179:96   0  7.2G  0 disk
└─mmcblk3p1  179:97   0  7.2G  0 part /mnt/sdnand

(3)缩小/dev/mmcblk0p8 分区(这里采用parted方法,也可以采用fdisk,具体见参考链接)

root@TQ3568:parted /dev/mmcblk0
GNU Parted 3.3
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) resizepart 8 25919487s  ##第8分区即/dev/mmcblk0p8
Warning: Shrinking a partition can cause data loss, are you sure you want to
continue?
Yes/No? yes
(parted) quit
Information: You may need to update /etc/fstab.

##(parted) resizepart 8 <新的结束位置>
##新的结束位置 = 起始位置 + 文件系统大小
##计算新的分区结束位置
##/dev/mmcblk0p8 起始扇区:13336576
##文件系统大小:6144M = 6 × 1024 × 1024 KB = 6291456 KB
##parted 的默认单位是 MB 或 GB,也可以用扇区 (s)
##可以用下面公式算结束扇区:
##结束扇区 = 起始扇区 + 文件系统大小(扇区数) - 1

##文件系统 6GB = 6 × 1024 × 1024 KB = 6291456 KB
##每个扇区 512B = 0.5 KB
##所以扇区数 = 6291456 KB ÷ 0.5 KB/扇区 = 12582912 扇区
##结束扇区 = 13336576 + 12582912 - 1 ≈ 25919487

(4)验证新文件系统

root@TQ3568:e2fsck -f /dev/mmcblk0p8
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
userdata: 16/835584 files (6.3% non-contiguous), 212293/6291456 blocks
root@TQ3568:
root@TQ3568:mount /dev/mmcblk0p8 /userdata   ##重新挂载到/userdata
root@TQ3568:
root@TQ3568:df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.9G  5.6G  2.1M 100% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           391M  1.5M  389M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           2.0G   16K  2.0G   1% /tmp
/dev/mmcblk0p7  119M   16M   96M  15% /oem
/dev/mmcblk3p1  7.3G   64K  7.3G   1% /mnt/sdnand
tmpfs           391M     0  391M   0% /run/user/0
tmpfs           391M   12K  391M   1% /run/user/1000
/dev/mmcblk0p8  5.8G  327K  5.7G   1% /userdata    ##缩容后文件系统的大小

(5)查看/dev/mmcblk0下未分配空间,可以看到刚才缩容的空间

root@TQ3568:parted /dev/mmcblk0 print free
Model: MMC 58A43A (sd/mmc)
Disk /dev/mmcblk0: 15.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name      Flags
        17.4kB  8389kB  8371kB  Free Space
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  83.9MB  67.1MB               boot
 4      83.9MB  218MB   134MB                recovery
 5      218MB   252MB   33.6MB               backup
 6      252MB   6694MB  6442MB  ext4         rootfs
 7      6694MB  6828MB  134MB   ext4         oem
 8      6828MB  13.3GB  6442MB  ext4         userdata
        13.3GB  15.7GB  2384MB  Free Space    ##未使用空间2348M

(6)扩容/dev/mmcblk0p6分区

这里遇到一个问题:空闲空间的start从13.3GB开始,如果扩容mmcblk0p6会和mmcblk0p7/8的分区重叠,导致扩容失败

(parted) resizepart 6 15.7GB
Error: Can't have overlapping partitions.  ##扩容失败

到这里打算先把oem和userdata中的内容先复制到/dev/mmcblk0p6分区当中(这里删除了一些日志文件还能有一些空余的空间),然后把mmcblk0p7/8分区先删除,等把mmcblk0p6扩容后再重新建立分区

##把/oem中的数据复制到/opt/tempdir/oem/oem
root@TQ3568:/opt/tempdir/oem/oem:ls -l
total 11976
-rw-r--r-- 1 root root   87402 Aug 19 14:21 200frames_count.h264
-rw-r--r-- 1 root root  512017 Aug 19 14:21 SampleJPGImage_500kbmb.jpg
-rw-r--r-- 1 root root 5253880 Aug 19 14:21 SampleVideo_1280x720_5mb.mp4
-rw-r--r-- 1 root root   81582 Aug 19 14:21 belle-nuit-testchart-1080p.png
drwxr-xr-x 3 root root    4096 Aug 19 14:21 dts
-rw-r--r-- 1 root root 7114880 Aug 19 14:21 game_test.gba
-rw-r--r-- 1 root root   12936 Aug 19 14:21 logo.bmp
-rw-r--r-- 1 root root   22364 Aug 19 14:21 logo_kernel.bmp
drwx------ 2 root root    4096 Aug 19 14:21 lost+found
-rw-r--r-- 1 root root  101760 Aug 19 14:21 piano2-CoolEdit.mp3
-rw-r--r-- 1 root root   96092 Aug 19 14:21 retroarch.cfg


##/userdata中的数据复制到/opt/tempdir/userdata
root@TQ3568:/opt/tempdir/userdata:ls -l
total 240
-rw-r--r-- 1 root root  87402 Aug 19 14:24 belle-nuit-testchart-1080p.png
drwx------ 2 root root   4096 Aug 19 14:24 lost+found
-rw-r--r-- 1 root root 101760 Aug 19 14:24 piano2-CoolEdit.mp3
-rwxr-xr-x 1 root root  48792 Aug 19 14:24 rtmpPush

取消mmcblk0p7/8分区的挂载

root@TQ3568:umount /dev/mmcblk0p7
root@TQ3568:umount /dev/mmcblk0p8
root@TQ3568:lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0      179:0    0 14.6G  0 disk
├─mmcblk0p1  179:1    0    4M  0 part
├─mmcblk0p2  179:2    0    4M  0 part
├─mmcblk0p3  179:3    0   64M  0 part
├─mmcblk0p4  179:4    0  128M  0 part
├─mmcblk0p5  179:5    0   32M  0 part
├─mmcblk0p6  179:6    0    6G  0 part /
├─mmcblk0p7  179:7    0  128M  0 part   ##取消挂载
└─mmcblk0p8  179:8    0    6G  0 part   ##取消挂载
mmcblk0boot0 179:32   0    4M  1 disk
mmcblk0boot1 179:64   0    4M  1 disk
mmcblk3      179:96   0  7.2G  0 disk
└─mmcblk3p1  179:97   0  7.2G  0 part /mnt/sdnand
root@TQ3568:

删除mmcblk0p7/8分区,并把mmcblk0p6分区扩容到末尾

(parted) print free
Model: MMC 58A43A (sd/mmc)
Disk /dev/mmcblk0: 15.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name      Flags
        17.4kB  8389kB  8371kB  Free Space
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  83.9MB  67.1MB               boot
 4      83.9MB  218MB   134MB                recovery
 5      218MB   252MB   33.6MB               backup
 6      252MB   6694MB  6442MB  ext4         rootfs
 7      6694MB  6828MB  134MB   ext4         oem
 8      6828MB  13.3GB  6442MB  ext4         userdata
        13.3GB  15.7GB  2384MB  Free Space

(parted)
(parted)
(parted)
(parted) rm 8   ##删除分区7/8
(parted) rm 7
(parted)
(parted) print free
Model: MMC 58A43A (sd/mmc)
Disk /dev/mmcblk0: 15.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name      Flags
        17.4kB  8389kB  8371kB  Free Space
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  83.9MB  67.1MB               boot
 4      83.9MB  218MB   134MB                recovery
 5      218MB   252MB   33.6MB               backup
 6      252MB   6694MB  6442MB  ext4         rootfs
        6694MB  15.7GB  8961MB  Free Space

(parted)
(parted) resizepart 6 100%   ##扩容到末尾
(parted)
(parted) print free
Model: MMC 58A43A (sd/mmc)
Disk /dev/mmcblk0: 15.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name      Flags
        17.4kB  8389kB  8371kB  Free Space
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  83.9MB  67.1MB               boot
 4      83.9MB  218MB   134MB                recovery
 5      218MB   252MB   33.6MB               backup
 6      252MB   15.7GB  15.4GB  ext4         rootfs   ##成功扩容mmcblk0p6分区

(parted)

(7)扩容mmcblk0p6上的文件系统/dev/root(这里偷个懒,不重新建mmcblk0p7/8分区了,里面的文件也不是很重要)

root@TQ3568:resize2fs /dev/mmcblk0p6  ##扩容文件系统
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/mmcblk0p6 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/mmcblk0p6 is now 3760635 (4k) blocks long.

root@TQ3568:
root@TQ3568:df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  5.5G  8.0G  41% /               ##扩容成功
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           391M  1.5M  389M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           2.0G   16K  2.0G   1% /tmp
/dev/mmcblk3p1  7.3G   64K  7.3G   1% /mnt/sdnand
tmpfs           391M   12K  391M   1% /run/user/1000
tmpfs           391M     0  391M   0% /run/user/0
root@TQ3568:

以上有参考文章缩小或扩大linux ext4文件系统(文末附上在线扩容根文件系统步骤) - 进取有乐 - 博客园

到此也解决了因根目录磁盘空间不足导致mysql启动失败的问题了。

root@TQ3568:systemctl start mysql
root@TQ3568:systemctl status mysql
● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset:
enabled)
     Active: active (running) since Tue 2025-08-19 14:45:16 CST; 10mi
n ago
    Process: 721 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exi
ted, status=0/SUCCESS)
   Main PID: 856 (mysqld)
     Status: "Server is operational"
     CGroup: /system.slice/mysql.service
             └─856 /usr/sbin/mysqld
root@TQ3568:

经过这次排查对Linux下的磁盘分区和文件系统有了一个新的认识。特别是学习如何对磁盘分区/文件系统的空间大小进行管理

更多推荐