As great as EBS volumes are on Amazon Web Services, they can break and not ever mount again, even though your data could still be there intact, a simple corruption on the filesystem structure can cause a lot of damage. On this post I teach you how to move all that data onto a new EBS drive, so keep calm and read slowly.<\/p>\n
So, you try to mount your drive after some updates and you get an error like this on dmesg | tail:<\/p>\n
so you unmount your drive, invoke xfs_repair and you get this…<\/p>\n attempting to find secondary superblock... and no good secondary superblock is found.<\/p>\n Don’t panic, this is what you have to do next to solve this issue:<\/p>\n Now here’s a gotcha, Amazon will not create your new drive using the same file system type (xfs), for some reason it will create it using the ext2 filesystem.<\/p>\n Here you have two options: or 2. copy the contents of your drive to a temporary location, usually inside \/mnt which has plenty of space from that ephemeral drive the ec2 instances come to, and then mkfs.xfs the new volume, and then copy the contents back… (which what I did, as I chose to create a larger drive and the ext2 format that came on the new volume only recognized the size of the snapshot)<\/p>\n Hope this saved your ass, leave a note if I did.<\/p>\n Remember to never do any irreversible action until you have a disk snapshot, try your best to never lose data.<\/p>\n","protected":false},"excerpt":{"rendered":" As great as EBS volumes are on Amazon Web Services, they can break and not ever mount again, even though your data could still be there intact, a simple corruption on the filesystem structure can cause a lot of damage. On this post I teach you how to move all that data onto a new […]<\/p>\n","protected":false},"author":1,"featured_media":3467,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[30,43],"tags":[1301,276,1158,1368,1449,1172,959,1366,1000,1369,1370,1367,1365],"class_list":["post-3213","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-geeklife","category-linux","tag-amazon-web-services","tag-cloud","tag-ebs","tag-filesystems","tag-linux","tag-server-administration","tag-sysadmin","tag-system-administration","tag-tips","tag-troubleshooting","tag-tutorials","tag-unix","tag-was"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/01\/car-longshot2.jpg?fit=720%2C482&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5Unzf-PP","jetpack-related-posts":[{"id":3370,"url":"https:\/\/www.gubatron.com\/blog\/how-to-resize-an-ebs-xfs-formatted-partition\/","url_meta":{"origin":3213,"position":0},"title":"How to resize an EBS (xfs formatted) partition","author":"gubatron","date":"December 20, 2014","format":false,"excerpt":"First of all, create a snapshot of your EBS volume. Then out of that snapshot you will be able to create your new volume. However, when you detach the old one from your instance and attach the new one, you will still see the old available space with df look\u2026","rel":"","context":"In "Code"","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2734,"url":"https:\/\/www.gubatron.com\/blog\/sample-etcfstab-entry-for-ebs-volume\/","url_meta":{"origin":3213,"position":1},"title":"Sample \/etc\/fstab entry for EBS volume","author":"gubatron","date":"May 23, 2012","format":false,"excerpt":"gid= and uid= are not valid for XFS, pass \"grpid=\" to your fstab entry. \/dev\/xvdf \/media\/ebs\/data xfs defaults,auto,noatime,noexec,grpid=1000 0 0","rel":"","context":"In \"aws\"","block_context":{"text":"aws","link":"https:\/\/www.gubatron.com\/blog\/tag\/aws\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3977,"url":"https:\/\/www.gubatron.com\/blog\/how-to-resize-aws-ec2-ebs-root-partition-without-rebooting-in-3-steps\/","url_meta":{"origin":3213,"position":2},"title":"How to resize AWS ec2 ebs root partition without rebooting in 3 steps","author":"gubatron","date":"September 14, 2022","format":false,"excerpt":"Go to the AWS EBS dashboard and modify the volume size. Might be good to create a snapshot of it for safety but haven't really failed ever doing this. # 1. Check the device of your partition $ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0\u2026","rel":"","context":"In "Code"","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":3127,"url":"https:\/\/www.gubatron.com\/blog\/cant-ssh-to-ec2-ubuntu-instance-etcfstab-breaks-bootup-due-to-missing-ebs-volume-solved\/","url_meta":{"origin":3213,"position":3},"title":"can’t ssh to ec2 ubuntu instance, \/etc\/fstab breaks bootup due to missing ebs volume [SOLVED]","author":"gubatron","date":"August 21, 2013","format":false,"excerpt":"So the \/etc\/fstab file on your root volume looked like this LABEL=cloudimg-rootfs \/ ext4 defaults 0 0 \/dev\/xvdf \/mnt\/backups auto defaults,comment=cloudconfig 0 2 by mistake you deleted the ebs volume that you had mounted on \/mnt\/backups (or whatever folder) and you restarted your ubuntu instance not knowing that if the\u2026","rel":"","context":"In "Code"","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2745,"url":"https:\/\/www.gubatron.com\/blog\/ubuntu-mongodb-wont-start-fixed\/","url_meta":{"origin":3213,"position":4},"title":"Ubuntu: mongodb won’t start [FIXED]","author":"gubatron","date":"May 23, 2012","format":false,"excerpt":"So you decided to change the default location of your mongodb data path on mongodb.conf but mongo won't start and you get an error similar to this [bash] Wed May 23 04:17:35 [initandlisten] MongoDB starting : pid=1214 port=27017 dbpath=\/media\/ebs\/data 64-bit host=domU-12-31-38-00-78-DB Wed May 23 04:17:35 [initandlisten] db version v2.0.4, pdfile\u2026","rel":"","context":"In \"mongo\"","block_context":{"text":"mongo","link":"https:\/\/www.gubatron.com\/blog\/tag\/mongo\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2989,"url":"https:\/\/www.gubatron.com\/blog\/raspberry-pi-game-console-project-getting-an-os-to-run-on-it\/","url_meta":{"origin":3213,"position":5},"title":"Raspberry PI Game Console Project: Getting an OS to run on it.","author":"gubatron","date":"April 10, 2013","format":false,"excerpt":"(My previous post) So now that we have a Raspberry PI we need an ARM based Operating system to go on it. My original idea is to run Jelly Bean on it, but I couldn't quickly find an image for the Raspberry PI (which means I'll either have to find\u2026","rel":"","context":"In "Gadgets"","block_context":{"text":"Gadgets","link":"https:\/\/www.gubatron.com\/blog\/category\/gadgets-2\/"},"img":{"alt_text":"8d8f488e9e5c11e2a3d822000a1f9be5_7","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2013\/04\/8d8f488e9e5c11e2a3d822000a1f9be5_7.jpg?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2013\/04\/8d8f488e9e5c11e2a3d822000a1f9be5_7.jpg?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2013\/04\/8d8f488e9e5c11e2a3d822000a1f9be5_7.jpg?resize=525%2C300 1.5x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/comments?post=3213"}],"version-history":[{"count":7,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3213\/revisions"}],"predecessor-version":[{"id":3468,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3213\/revisions\/3468"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/media\/3467"}],"wp:attachment":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/media?parent=3213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/categories?post=3213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/tags?post=3213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}[56439860.329754] XFS (xvdf): Corruption detected. Unmount and run xfs_repair<\/code><\/p>\n$ sudo xfs_repair -n \/dev\/xvdf
\nPhase 1 - find and verify superblock...
\nbad primary superblock - bad magic number !!!<\/p>\n
\n..........................................<\/code><\/p>\n\n
$ sudo file -s \/dev\/xvdg
\n\/dev\/xvdg: Linux rev 1.0 ext2 filesystem data (mounted or unclean), UUID=2e35874f-1d21-4d2d-b42b-ae27966e0aab (large files)<\/code><\/p>\n
\n1. Live with the new ext2 file system, make sure your \/etc\/fstab is updated to look something like this:
\n\/dev\/xvdg \/path\/to\/mount\/to auto defaults,nobootwait,noatime 0 0<\/code><\/p>\n