OT:Question on NVME disk direct access?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

OT:Question on NVME disk direct access?

Michael D. Setzer II-2
I'm the maintainer of the G4L disk imaging project since 2004, and I use
Fedora as the build platform. Currently using Fedora 27.

Have an issue from a user that has me baffled, so am hoping someone here
might provide some guidance.

The program boots a linux kernel, and basically uses dd to copy the
disk/partitions thru a compression program and creates an image file on ftp
server or local device.

I don't have any physical nvme disks, but using virtualbox I created a 4M
disk, and 2 - 2M partitions within it. In testing that, the 4M disk compresses to
a 30K file, and the 2M partitions compress to about 15K each. That is what is
expected with cleared partitions.

The user though, with a real 256G disk doesn't seem to get any compression
of the disk or partitions. Them resulting images are close to the same size as
the disks or partitions??

He can mount the partitions and see the files, so there must be something
going on that I don't see?

Would think that accessing the /dev/nvme0n1 or partitions /dev/nvme0n1p1
thru p5 would act the same as accessing /dev/sda or /dev/sdax partitions.

The images that are created pass the compression program test, so it is
reading data, but in some form that doesn't compress much, and user has
used a program to clear the unused space?

Thanks for your time, and any ideals.

+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65094199.902703 | ABC          16613838.513356
SETI        109335656.779734 | EINSTEIN    140992828.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Gordon Messmer-2
On 04/02/2018 11:30 AM, Michael D. Setzer II wrote:
> The user though, with a real 256G disk doesn't seem to get any compression
> of the disk or partitions. Them resulting images are close to the same size as
> the disks or partitions??


Does the user have LUKS encrypted partitions?  Encrypting the system is
trivial, and I would imagine common.  If the user had selected the
option to encrypt their system, a direct disk image would compress very
little.

_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
In reply to this post by Michael D. Setzer II-2


On 04/02/2018 11:30 AM, Michael D. Setzer II wrote:

> I'm the maintainer of the G4L disk imaging project since 2004, and I use
> Fedora as the build platform. Currently using Fedora 27.
>
> Have an issue from a user that has me baffled, so am hoping someone here
> might provide some guidance.
>
> The program boots a linux kernel, and basically uses dd to copy the
> disk/partitions thru a compression program and creates an image file on ftp
> server or local device.
>
> I don't have any physical nvme disks, but using virtualbox I created a 4M
> disk, and 2 - 2M partitions within it. In testing that, the 4M disk compresses to
> a 30K file, and the 2M partitions compress to about 15K each. That is what is
> expected with cleared partitions.
>
> The user though, with a real 256G disk doesn't seem to get any compression
> of the disk or partitions. Them resulting images are close to the same size as
> the disks or partitions??
>
> He can mount the partitions and see the files, so there must be something
> going on that I don't see?
>
> Would think that accessing the /dev/nvme0n1 or partitions /dev/nvme0n1p1
> thru p5 would act the same as accessing /dev/sda or /dev/sdax partitions.
>
> The images that are created pass the compression program test, so it is
> reading data, but in some form that doesn't compress much, and user has
> used a program to clear the unused space?
>
> Thanks for your time, and any ideals.

Hi Michael,

This probably won't help as I don't entirely understand your
question.

My FC27 system has a LUKS encrypted 1 GB NVMe drive.  I clone
the drive to a mechanical drive as a poor man's RAID1.  NVMe
drives don't do RAID1.

My first attempt, was booting off a Live USB and do a "dd".
It was a disaster.  Took 14 hours and did not work in the
end.

Then I switched Clone Zilla to do the clone and it has worked
perfectly about 5 times now.  The mechanical clone drive boots
perfectly too.

Clone Zilla only takes 1:24 to clone.  It uses dd to clone LUKS
drives, so who knows why Clone Zilla's works and mine does not.

-T
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
On 4 Apr 2018 at 21:01, Todd Chester wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             [hidden email]
From:           Todd Chester <[hidden email]>
Date sent:       Wed, 4 Apr 2018 21:01:15 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

>
>
> On 04/02/2018 11:30 AM, Michael D. Setzer II wrote:
> > I'm the maintainer of the G4L disk imaging project since 2004, and I use
> > Fedora as the build platform. Currently using Fedora 27.
> >
> > Have an issue from a user that has me baffled, so am hoping someone here
> > might provide some guidance.
> >
> > The program boots a linux kernel, and basically uses dd to copy the
> > disk/partitions thru a compression program and creates an image file on ftp
> > server or local device.
> >
> > I don't have any physical nvme disks, but using virtualbox I created a 4M
> > disk, and 2 - 2M partitions within it. In testing that, the 4M disk compresses to
> > a 30K file, and the 2M partitions compress to about 15K each. That is what is
> > expected with cleared partitions.
> >
> > The user though, with a real 256G disk doesn't seem to get any compression
> > of the disk or partitions. Them resulting images are close to the same size as
> > the disks or partitions??
> >
> > He can mount the partitions and see the files, so there must be something
> > going on that I don't see?
> >
> > Would think that accessing the /dev/nvme0n1 or partitions /dev/nvme0n1p1
> > thru p5 would act the same as accessing /dev/sda or /dev/sdax partitions.
> >
> > The images that are created pass the compression program test, so it is
> > reading data, but in some form that doesn't compress much, and user has
> > used a program to clear the unused space?
> >
> > Thanks for your time, and any ideals.
>
> Hi Michael,
>
> This probably won't help as I don't entirely understand your
> question.
>
> My FC27 system has a LUKS encrypted 1 GB NVMe drive.  I clone
> the drive to a mechanical drive as a poor man's RAID1.  NVMe
> drives don't do RAID1.
>
> My first attempt, was booting off a Live USB and do a "dd".
> It was a disaster.  Took 14 hours and did not work in the
> end.
>
> Then I switched Clone Zilla to do the clone and it has worked
> perfectly about 5 times now.  The mechanical clone drive boots
> perfectly too.
>
> Clone Zilla only takes 1:24 to clone.  It uses dd to clone LUKS
> drives, so who knows why Clone Zilla's works and mine does not.
>
> -T
> ________________________________

Thanks for the reply. There are lots of issues with doing cloning.
Usually, doing a disk clone gets arround issues where the boot loader is
using the blkid, since it makes the blikids for the partitions the same. Problem
with that thou is that you can't have to disks in the same machine with the
same blkids. Once cloned a disk, and then rebooted it to the OS without
disconnecting, and for some reason, it mounted some partitions from the first
disk, and others from the second?

Same issue with the boot loaders using the /dev/sdx option. If you clone a
disk on /dev/sda to /dev/sdb it works fine, but if you remove sda to test if it
will boot, it will not since second disk with have sdb instead of the sda. Have
to switch cables, or change boot order in bios.

Contacted the person in charge of the nvme program, and he says it should
work as it does with the virtualbox test I did.

User was using a windows program called eraser to clear the drive, but from
what I have just found it seems to be a security eraser, and rights random
data to the unused space as contrasted to writing nulls. Think the program
was probable working just fine, but with completely random data the lzop
compression doesn't work well. About twice the speed of gzip but 10% larger
images. I could take a 1T disk, and compress it down to a 40G file with
Windows 10 and Fedora 25 on it.

My classroom setup also, had and NFTS clone image file on a separate
partitions, and had an grub boot option, that would reimage the 160G
Windows 10 partition in about 12 minutes.  About a 20G image file.

Have a program on the g4l disk that will zero out the unused space, so have
asked the user to try using that to clean disk, and then make image.

Hopefully, that will result in the expected compression.

Thanks again for the info.

_______________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65132161.613216 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141004554.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo


On 04/04/2018 09:43 PM, Michael D. Setzer II wrote:

> Thanks for the reply. There are lots of issues with doing cloning.
> Usually, doing a disk clone gets arround issues where the boot loader is
> using the blkid, since it makes the blikids for the partitions the same. Problem
> with that thou is that you can't have to disks in the same machine with the
> same blkids. Once cloned a disk, and then rebooted it to the OS without
> disconnecting, and for some reason, it mounted some partitions from the first
> disk, and others from the second?
>
> Same issue with the boot loaders using the /dev/sdx option. If you clone a
> disk on /dev/sda to /dev/sdb it works fine, but if you remove sda to test if it
> will boot, it will not since second disk with have sdb instead of the sda. Have
> to switch cables, or change boot order in bios.
>
> Contacted the person in charge of the nvme program, and he says it should
> work as it does with the virtualbox test I did.

Chuckle.  I have the same problem.  My two backup drives and my clone
drive fit into a removable SATA drive sleeve.  If I forget to remove
the clone drive after I do the clone, Fedora will boot off the clone
drive and mix things up as you describe.  I researched as to why
this happens and it is all do to both drives having the same UUID
numbers on their partitions.

I am not in my office at the moment, but when I do get back, I can
send you a bash script I wrote to warm me when this happens.  Let
me know if you would like it.

I frequently forget to swap out the clone drive and reinsert one
of the backup drives.  The first clue is that it takes about
eight times longer to boot.


> User was using a windows program called eraser to clear the drive, but from
> what I have just found it seems to be a security eraser, and rights random
> data to the unused space as contrasted to writing nulls. Think the program
> was probable working just fine, but with completely random data the lzop
> compression doesn't work well. About twice the speed of gzip but 10% larger
> images. I could take a 1T disk, and compress it down to a 40G file with
> Windows 10 and Fedora 25 on it.

If you what to completely blank a disk out, don't mess around with that
operating system whose name I shall not mention.  (I hear it is slow,
buggy, and expensive.)  Stay in Linux.

    # dd bs=1M if=/dev/zero of=/dev/sdx

/dev/zero is the fastest, but you also have /dev/one which is slower
and /dev/random which is really slow.


> My classroom setup also, had and NFTS clone image file on a separate
> partitions, and had an grub boot option, that would reimage the 160G
> Windows 10 partition in about 12 minutes.  About a 20G image file.
>
> Have a program on the g4l disk that will zero out the unused space, so have
> asked the user to try using that to clean disk, and then make image.
>
> Hopefully, that will result in the expected compression.

Take a look at Clone Zilla, it does all that for you:
http://clonezilla.org/

The author is extremely responsive to questions too.

I use Clone Zilla with its rescue mode (advanced setting) on
NTFS drives with bad sectors on them ALL THE TIME.

What ?? Yes, I have to work on THAT operating system whose name I
shall not mentions too.  My customer base is mainly small
businesses and they can not get apps to run on anything other
than THAT operating system.  It is what it is.

Tip: stay the hell away from Intel SSD drives.  I use to sell them.
They are garbage.  I took around a $2000 loss so far having to replace
them as they go bad in my customer's machines.

I switched to Samsung SSD's and they are rock solid.  Not a
single failure yet.  They are about 20% more expensive than
Intel's drives, but when your have to replace them for free
and just before they brick and lose your customer's data,
they become extremely expensive.

Samsung's NVMe drives are awesome.  (My wife named them NeVada
Medical Examiner drives to remember the abbreviation.)

Tip: try to size your SSD drives such that they have at least 50%
or more free space on them.  This improves your wear life.  And
the wear leveling algorithm on the drive will adore you.


>
> Thanks again for the info.

You are most welcome.

-T
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
My G4L program can clone to local disk or usb flash. Use a 128G USB flash
for my home machine, and to do my mom's machine as a backup.

If hard disk were to crash, could quickly pop out the disk, and put in a new
one, then boot from the usb flash or cd, and reimage.

At the College, I had an ftp server in my classroom that was running 1G
network, and would make a compressed image on the server. Could even to
a PXE boot from machines to load the g4l and do an image. Even had g4l as
a boot option in the grub menu, just had to modify the 40_custom file, and
put the kernel file and ramdisk.lzma in the /boot directory.

Have a script on the g4l image that can clean partitions, and it supports linux,
ntfs, fat32 and even swap partitions. With fat32, it has to make multiple 2G
files until disk is full due to the file size limit. Other ones don't have that issue,
but fat32 is getting less and less common.

Use lzop as compression program, since it is about twice as fast as gzip, and
image is only about 10% larger.

Program is on sourceforge, and is free, and has full source code if one wants
to look at it.



On 5 Apr 2018 at 0:59, Todd Chester wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             [hidden email]
From:           Todd Chester <[hidden email]>
Date sent:       Thu, 5 Apr 2018 00:59:39 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

>
>
> On 04/04/2018 09:43 PM, Michael D. Setzer II wrote:
>
> > Thanks for the reply. There are lots of issues with doing cloning.
> > Usually, doing a disk clone gets arround issues where the boot loader is
> > using the blkid, since it makes the blikids for the partitions the same. Problem
> > with that thou is that you can't have to disks in the same machine with the
> > same blkids. Once cloned a disk, and then rebooted it to the OS without
> > disconnecting, and for some reason, it mounted some partitions from the first
> > disk, and others from the second?
> >
> > Same issue with the boot loaders using the /dev/sdx option. If you clone a
> > disk on /dev/sda to /dev/sdb it works fine, but if you remove sda to test if it
> > will boot, it will not since second disk with have sdb instead of the sda. Have
> > to switch cables, or change boot order in bios.
> >
> > Contacted the person in charge of the nvme program, and he says it should
> > work as it does with the virtualbox test I did.
>
> Chuckle.  I have the same problem.  My two backup drives and my clone
> drive fit into a removable SATA drive sleeve.  If I forget to remove
> the clone drive after I do the clone, Fedora will boot off the clone
> drive and mix things up as you describe.  I researched as to why
> this happens and it is all do to both drives having the same UUID
> numbers on their partitions.
>
> I am not in my office at the moment, but when I do get back, I can
> send you a bash script I wrote to warm me when this happens.  Let
> me know if you would like it.
>
> I frequently forget to swap out the clone drive and reinsert one
> of the backup drives.  The first clue is that it takes about
> eight times longer to boot.
>
>
> > User was using a windows program called eraser to clear the drive, but from
> > what I have just found it seems to be a security eraser, and rights random
> > data to the unused space as contrasted to writing nulls. Think the program
> > was probable working just fine, but with completely random data the lzop
> > compression doesn't work well. About twice the speed of gzip but 10% larger
> > images. I could take a 1T disk, and compress it down to a 40G file with
> > Windows 10 and Fedora 25 on it.
>
> If you what to completely blank a disk out, don't mess around with that
> operating system whose name I shall not mention.  (I hear it is slow,
> buggy, and expensive.)  Stay in Linux.
>
>     # dd bs=1M if=/dev/zero of=/dev/sdx
>
> /dev/zero is the fastest, but you also have /dev/one which is slower
> and /dev/random which is really slow.
>
>
> > My classroom setup also, had and NFTS clone image file on a separate
> > partitions, and had an grub boot option, that would reimage the 160G
> > Windows 10 partition in about 12 minutes.  About a 20G image file.
> >
> > Have a program on the g4l disk that will zero out the unused space, so have
> > asked the user to try using that to clean disk, and then make image.
> >
> > Hopefully, that will result in the expected compression.
>
> Take a look at Clone Zilla, it does all that for you:
> http://clonezilla.org/
>
> The author is extremely responsive to questions too.
>
> I use Clone Zilla with its rescue mode (advanced setting) on
> NTFS drives with bad sectors on them ALL THE TIME.
>
> What ?? Yes, I have to work on THAT operating system whose name I
> shall not mentions too.  My customer base is mainly small
> businesses and they can not get apps to run on anything other
> than THAT operating system.  It is what it is.
>
> Tip: stay the hell away from Intel SSD drives.  I use to sell them.
> They are garbage.  I took around a $2000 loss so far having to replace
> them as they go bad in my customer's machines.
>
> I switched to Samsung SSD's and they are rock solid.  Not a
> single failure yet.  They are about 20% more expensive than
> Intel's drives, but when your have to replace them for free
> and just before they brick and lose your customer's data,
> they become extremely expensive.
>
> Samsung's NVMe drives are awesome.  (My wife named them NeVada
> Medical Examiner drives to remember the abbreviation.)
>
> Tip: try to size your SSD drives such that they have at least 50%
> or more free space on them.  This improves your wear life.  And
> the wear leveling algorithm on the drive will adore you.
>
>
> >
> > Thanks again for the info.
>
> You are most welcome.
>
> -T
> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65132161.613216 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141004554.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Samuel Sieb
In reply to this post by ToddAndMargo
On 04/05/2018 12:59 AM, Todd Chester wrote:
> If you what to completely blank a disk out, don't mess around with that
> operating system whose name I shall not mention.  (I hear it is slow,
> buggy, and expensive.)  Stay in Linux.
>
>     # dd bs=1M if=/dev/zero of=/dev/sdx

If I understood the original post correctly, the person was wanting to
blank the free space, not erase the whole disk.  The problem was that
the program used was a secure eraser which filled the blank space with
random data which doesn't compress.
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
On 04/05/2018 08:36 AM, Samuel Sieb wrote:

> On 04/05/2018 12:59 AM, Todd Chester wrote:
>> If you what to completely blank a disk out, don't mess around with that
>> operating system whose name I shall not mention.  (I hear it is slow,
>> buggy, and expensive.)  Stay in Linux.
>>
>>     # dd bs=1M if=/dev/zero of=/dev/sdx
>
> If I understood the original post correctly, the person was wanting to
> blank the free space, not erase the whole disk.  The problem was that
> the program used was a secure eraser which filled the blank space with
> random data which doesn't compress.

CLone Zilla skips over data that is not allocated, unless
your are LUKS encrypted.  Goes like the wind!
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
In reply to this post by Michael D. Setzer II-2


On 04/04/2018 09:43 PM, Michael D. Setzer II wrote:

> Thanks for the reply. There are lots of issues with doing cloning.
> Usually, doing a disk clone gets arround issues where the boot loader is
> using the blkid, since it makes the blikids for the partitions the same. Problem
> with that thou is that you can't have to disks in the same machine with the
> same blkids. Once cloned a disk, and then rebooted it to the OS without
> disconnecting, and for some reason, it mounted some partitions from the first
> disk, and others from the second?
>
> Same issue with the boot loaders using the /dev/sdx option. If you clone a
> disk on /dev/sda to /dev/sdb it works fine, but if you remove sda to test if it
> will boot, it will not since second disk with have sdb instead of the sda. Have
> to switch cables, or change boot order in bios.
>
> Contacted the person in charge of the nvme program, and he says it should
> work as it does with the virtualbox test I did.

Chuckle.  I have the same problem.  My two backup drives and my clone
drive fit into a removable SATA drive sleeve.  If I forget to remove
the clone drive after I do the clone, Fedora will boot off the clone
drive and mix things up as you describe.  I researched as to why
this happens and it is all do to both drives having the same UUID
numbers on their partitions.

I am not in my office at the moment, but when I do get back, I can
send you a bash script I wrote to warm me when this happens.  Let
me know if you would like it.

I frequently forget to swap out the clone drive and reinsert one
of the backup drives.  The first clue is that it takes about
eight times longer to boot.


> User was using a windows program called eraser to clear the drive, but from
> what I have just found it seems to be a security eraser, and rights random
> data to the unused space as contrasted to writing nulls. Think the program
> was probable working just fine, but with completely random data the lzop
> compression doesn't work well. About twice the speed of gzip but 10% larger
> images. I could take a 1T disk, and compress it down to a 40G file with
> Windows 10 and Fedora 25 on it.

If you what to completely blank a disk out, don't mess around with that
operating system whose name I shall not mention.  (I hear it is slow,
buggy, and expensive.)  Stay in Linux.

    # dd bs=1M if=/dev/zero of=/dev/sdx

/dev/zero is the fastest, but you also have /dev/one which is slower
and /dev/random which is really slow.


> My classroom setup also, had and NFTS clone image file on a separate
> partitions, and had an grub boot option, that would reimage the 160G
> Windows 10 partition in about 12 minutes.  About a 20G image file.
>
> Have a program on the g4l disk that will zero out the unused space, so have
> asked the user to try using that to clean disk, and then make image.
>
> Hopefully, that will result in the expected compression.

Take a look at Clone Zilla, it does all that for you:
http://clonezilla.org/

The author is extremely responsive to questions too.

I use Clone Zilla with its rescue mode (advanced setting) on
NTFS drives with bad sectors on them ALL THE TIME.

What ?? Yes, I have to work on THAT operating system whose name I
shall not mentions too.  My customer base is mainly small
businesses and they can not get apps to run on anything other
than THAT operating system.  It is what it is.

Tip: stay the hell away from Intel SSD drives.  I use to sell them.
They are garbage.  I took around a $2000 loss so far having to replace
them as they go bad in my customer's machines.

I switched to Samsung SSD's and they are rock solid.  Not a
single failure yet.  They are about 20% more expensive than
Intel's drives, but when your have to replace them for free
and just before they brick and lose your customer's data,
they become extremely expensive.

Samsung's NVMe drives are awesome.  (My wife named them NeVada
Medical Examiner drives to remember the abbreviation.)

Tip: try to size your SSD drives such that they have at least 50%
or more free space on them.  This improves your wear life.  And
the wear leveling algorithm on the drive will adore you.


>
> Thanks again for the info.

You are most welcome.

-T
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
I like making image files, and include the date as back of the image name.
That way, I can have multiple backups of images, and in the event of
something going wrong, can restore older versions on other disk to recover
files that might have been deleted that should not have been.


On 5 Apr 2018 at 10:21, ToddAndMargo wrote:

From:           ToddAndMargo <[hidden email]>
Subject:         Re: OT:Question on NVME disk direct access?
To:             [hidden email], [hidden email]
Date sent:       Thu, 5 Apr 2018 10:21:26 -0700

>
>
> On 04/04/2018 09:43 PM, Michael D. Setzer II wrote:
>
> > Thanks for the reply. There are lots of issues with doing cloning.
> > Usually, doing a disk clone gets arround issues where the boot loader is
> > using the blkid, since it makes the blikids for the partitions the same. Problem
> > with that thou is that you can't have to disks in the same machine with the
> > same blkids. Once cloned a disk, and then rebooted it to the OS without
> > disconnecting, and for some reason, it mounted some partitions from the first
> > disk, and others from the second?
> >
> > Same issue with the boot loaders using the /dev/sdx option. If you clone a
> > disk on /dev/sda to /dev/sdb it works fine, but if you remove sda to test if it
> > will boot, it will not since second disk with have sdb instead of the sda. Have
> > to switch cables, or change boot order in bios.
> >
> > Contacted the person in charge of the nvme program, and he says it should
> > work as it does with the virtualbox test I did.
>
> Chuckle.  I have the same problem.  My two backup drives and my clone
> drive fit into a removable SATA drive sleeve.  If I forget to remove
> the clone drive after I do the clone, Fedora will boot off the clone
> drive and mix things up as you describe.  I researched as to why
> this happens and it is all do to both drives having the same UUID
> numbers on their partitions.
>
> I am not in my office at the moment, but when I do get back, I can
> send you a bash script I wrote to warm me when this happens.  Let
> me know if you would like it.
>
> I frequently forget to swap out the clone drive and reinsert one
> of the backup drives.  The first clue is that it takes about
> eight times longer to boot.
>
>
> > User was using a windows program called eraser to clear the drive, but from
> > what I have just found it seems to be a security eraser, and rights random
> > data to the unused space as contrasted to writing nulls. Think the program
> > was probable working just fine, but with completely random data the lzop
> > compression doesn't work well. About twice the speed of gzip but 10% larger
> > images. I could take a 1T disk, and compress it down to a 40G file with
> > Windows 10 and Fedora 25 on it.
>
> If you what to completely blank a disk out, don't mess around with that
> operating system whose name I shall not mention.  (I hear it is slow,
> buggy, and expensive.)  Stay in Linux.
>
>     # dd bs=1M if=/dev/zero of=/dev/sdx
>
> /dev/zero is the fastest, but you also have /dev/one which is slower
> and /dev/random which is really slow.
>
>
> > My classroom setup also, had and NFTS clone image file on a separate
> > partitions, and had an grub boot option, that would reimage the 160G
> > Windows 10 partition in about 12 minutes.  About a 20G image file.
> >
> > Have a program on the g4l disk that will zero out the unused space, so have
> > asked the user to try using that to clean disk, and then make image.
> >
> > Hopefully, that will result in the expected compression.
>
> Take a look at Clone Zilla, it does all that for you:
> http://clonezilla.org/
>
> The author is extremely responsive to questions too.
>
> I use Clone Zilla with its rescue mode (advanced setting) on
> NTFS drives with bad sectors on them ALL THE TIME.
>
> What ?? Yes, I have to work on THAT operating system whose name I
> shall not mentions too.  My customer base is mainly small
> businesses and they can not get apps to run on anything other
> than THAT operating system.  It is what it is.
>
> Tip: stay the hell away from Intel SSD drives.  I use to sell them.
> They are garbage.  I took around a $2000 loss so far having to replace
> them as they go bad in my customer's machines.
>
> I switched to Samsung SSD's and they are rock solid.  Not a
> single failure yet.  They are about 20% more expensive than
> Intel's drives, but when your have to replace them for free
> and just before they brick and lose your customer's data,
> they become extremely expensive.
>
> Samsung's NVMe drives are awesome.  (My wife named them NeVada
> Medical Examiner drives to remember the abbreviation.)
>
> Tip: try to size your SSD drives such that they have at least 50%
> or more free space on them.  This improves your wear life.  And
> the wear leveling algorithm on the drive will adore you.
>
>
> >
> > Thanks again for the info.
>
> You are most welcome.
>
> -T
>


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65145396.630944 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141006633.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
In reply to this post by ToddAndMargo
On 5 Apr 2018 at 9:58, ToddAndMargo wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             [hidden email]
From:           ToddAndMargo <[hidden email]>
Date sent:       Thu, 5 Apr 2018 09:58:34 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

> On 04/05/2018 08:36 AM, Samuel Sieb wrote:
> > On 04/05/2018 12:59 AM, Todd Chester wrote:
> >> If you what to completely blank a disk out, don't mess around with that
> >> operating system whose name I shall not mention.  (I hear it is slow,
> >> buggy, and expensive.)  Stay in Linux.
> >>
> >>     # dd bs=1M if=/dev/zero of=/dev/sdx
> >
> > If I understood the original post correctly, the person was wanting to
> > blank the free space, not erase the whole disk.  The problem was that
> > the program used was a secure eraser which filled the blank space with
> > random data which doesn't compress.
>
> CLone Zilla skips over data that is not allocated, unless
> your are LUKS encrypted.  Goes like the wind!

There are various options with programs. At the bit level imaging for a bare
image restore, the OS of the disk are partition is not used, so it is just reading
the raw data of all the sectors, so that a single pass restore gives everything.
If one uses an option to access the data at the OS level, it can determine
what is used or not. G4L includes ntfsclone for windows paritions, and
fsarchiver for that and others, and it can backup data only, and is faster.
Also, has options to change size of partitions. Bit level restores exactly the
same size, but can be resized by other utilities afterwards.


> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65145396.630944 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141006633.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
In reply to this post by Samuel Sieb
On 5 Apr 2018 at 8:36, Samuel Sieb wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             [hidden email]
From:           Samuel Sieb <[hidden email]>
Date sent:       Thu, 5 Apr 2018 08:36:20 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

> On 04/05/2018 12:59 AM, Todd Chester wrote:
> > If you what to completely blank a disk out, don't mess around with that
> > operating system whose name I shall not mention.  (I hear it is slow,
> > buggy, and expensive.)  Stay in Linux.
> >
> >     # dd bs=1M if=/dev/zero of=/dev/sdx
>
> If I understood the original post correctly, the person was wanting to
> blank the free space, not erase the whole disk.  The problem was that
> the program used was a secure eraser which filled the blank space with
> random data which doesn't compress.

It has now been confirmed that is the case. The user has used the G4L
included cleaning options for the free space on the partitions, and is seeing a
compression rate of close to 90%. The Eraser programs current
documentation online seems to show default option is a security erase rather
than a clearing out with zeroing. Program does have an option for that, but
not default.

G4L creates a 0bits file that it writes nulls to, until the partition is full and then
deletes file. With Fat32 partitions, it creats multiple 2G files till partition full,
and then deletes them. It doesn't blank out disks or partitions. Exception is
for swap partition of linux, where it does get blkid info of partition, then zeros
out partitions, and recreates the swap partition with same blkid info.

Thanks.


> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65145396.630944 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141006633.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
In reply to this post by Michael D. Setzer II-2


On 04/05/2018 01:15 PM, Michael D. Setzer II wrote:
> I like making image files, and include the date as back of the image name.
> That way, I can have multiple backups of images, and in the event of
> something going wrong, can restore older versions on other disk to recover
> files that might have been deleted that should not have been.

CloneZilla will do a disk to image, including a network location
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
In reply to this post by Michael D. Setzer II-2


On 04/05/2018 01:15 PM, Michael D. Setzer II wrote:
> G4L includes ntfsclone for windows paritions, and
> fsarchiver for that and others, and it can backup data only, and is faster.
> Also, has options to change size of partitions. Bit level restores exactly the
> same size, but can be resized by other utilities afterwards.

Interesting.  I am going to have to try out G4L.  CloneZilla will
not do resizing.
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
In reply to this post by ToddAndMargo
G4L does a few things. It has UDPcast, where it can transfer an image to
multiple machines at one time.  Would create image files on one system, and
the broadcast it to the 19 other machines at one time. Has other little things.

Had students work on documentation file, and the latest is at
https://sourceforge.net/projects/g4l/files/g4l%20documentation/g4l0.53-docu
mentation.pdf/download

That was built using Fedora 24 as a base. THe 0.54 version is now using
Fedora 27 as the build system.


On 5 Apr 2018 at 21:38, Todd Chester wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             "Michael D. Setzer II" <[hidden email]>,
        Community support for Fedora users
<[hidden email]>
From:           Todd Chester <[hidden email]>
Date sent:       Thu, 5 Apr 2018 21:38:10 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

>
>
> On 04/05/2018 01:15 PM, Michael D. Setzer II wrote:
> > I like making image files, and include the date as back of the image name.
> > That way, I can have multiple backups of images, and in the event of
> > something going wrong, can restore older versions on other disk to recover
> > files that might have been deleted that should not have been.
>
> CloneZilla will do a disk to image, including a network location
> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65145396.630944 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141006633.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

Michael D. Setzer II-2
In reply to this post by ToddAndMargo
The raw mode does not do resizing, but with ntfsclone you can restore and
image to a larger partition. At that point, it will still be the same size as the
original partition, but then you run the ntfsclone resize option, and it will than
modify the partition and can make use of the additional space. It could also,
reduce the size, but have not done that myself.

fsarchiver also can do this, but it is not a program I wrote, but was asked to
include long ago. It is a file level image, so partitions need to already exist.


On 5 Apr 2018 at 21:41, Todd Chester wrote:

Subject:         Re: OT:Question on NVME disk direct access?
To:             "Michael D. Setzer II" <[hidden email]>,
        Community support for Fedora users
<[hidden email]>
From:           Todd Chester <[hidden email]>
Date sent:       Thu, 5 Apr 2018 21:41:23 -0700
Send reply to:   Community support for Fedora users
<[hidden email]>

>
>
> On 04/05/2018 01:15 PM, Michael D. Setzer II wrote:
> > G4L includes ntfsclone for windows paritions, and
> > fsarchiver for that and others, and it can backup data only, and is faster.
> > Also, has options to change size of partitions. Bit level restores exactly the
> > same size, but can be resized by other utilities afterwards.
>
> Interesting.  I am going to have to try out G4L.  CloneZilla will
> not do resizing.
> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)    
 mailto:[hidden email]                            
 mailto:[hidden email]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA      65145396.630944 | ABC          16613838.513356
SETI        109336776.035164 | EINSTEIN    141006633.999240
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: OT:Question on NVME disk direct access?

ToddAndMargo
On 04/05/2018 11:21 PM, Michael D. Setzer II wrote:
> The raw mode does not do resizing, but with ntfsclone you can restore and
> image to a larger partition. At that point, it will still be the same size as the
> original partition, but then you run the ntfsclone resize option, and it will than
> modify the partition and can make use of the additional space. It could also,
> reduce the size, but have not done that myself.

I use gparted off the Xfce Live USB to resize.
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]