Forums

How to embedd native storage protocoles in h/w appliance?

Started by rohit September 10, 2006
Dear All,

(Note: I am a newbie to embedded system programming (hardware
programming), so forgive me if it seems a silly query.)

I need to develop a hardware appliance that will be deployed in the
data path between clients and the storage device (for eg. harddisk).
This hardware box should be able to understand the native storage
protocol (eg. SCSI) over which it is travelling , get the data, perform
some operations on the data (eg. mangle the data) and send forward to
the harddisk.

So my query is:
1) How to get started for developing such hardware appliance ? Do we
need to program the logic for it in ASSEMBLY language and burn it into
the eeprom ? Or there are some alternative methods to achieve the same?

2) What are the possible tools available for such types of hardware
programming ?

3) Basically, I need to embedded the protocol specification into my
hardware. Is that practically feasible ? Or I am going in the wrong
direction. Please guide.

Any pointers, references, comments will be of great help.

Thanks and Regards,
Rohit Dhamija

On 10 Sep 2006 09:12:29 -0700, "rohit" <rohit.dhamija@gmail.com> wrote
in comp.arch.embedded:

> Dear All, > > (Note: I am a newbie to embedded system programming (hardware > programming), so forgive me if it seems a silly query.) > > I need to develop a hardware appliance that will be deployed in the
The first question that I need to ask is what is your definition of a "hardware appliance"? Most hardware appliances today have one or more processors in them (microprocessor, microcontroller, DSP). Or do you mean that you are trying to do this all in hardware, such as in an FPGA?
> data path between clients and the storage device (for eg. harddisk). > This hardware box should be able to understand the native storage > protocol (eg. SCSI) over which it is travelling , get the data, perform > some operations on the data (eg. mangle the data) and send forward to > the harddisk.
Is this some sort of data protection tool? Are you trying to encrypt on a file-by-file basis, or an a low level block basis?
> So my query is: > 1) How to get started for developing such hardware appliance ? Do we > need to program the logic for it in ASSEMBLY language and burn it into > the eeprom ? Or there are some alternative methods to achieve the same?
Anything that you can program in an assembly language, you can program in a higher level language.
> 2) What are the possible tools available for such types of hardware > programming ? > > 3) Basically, I need to embedded the protocol specification into my > hardware. Is that practically feasible ? Or I am going in the wrong > direction. Please guide. > > Any pointers, references, comments will be of great help. > > Thanks and Regards, > Rohit Dhamija
I think that your chances of getting useful answers will improve greatly if you provide more information about what you are trying to do. And what the term "hardware appliance" means to you and your company and your customers. Most embedded systems could be asked as "hardware appliances", because usually they don't operate with keyboards, mice, and video displays. There is a router on my desk that connects my computers to a cable modem. I imaging that most people would classify it as a "hardware appliance". It's only visible interface is half a dozen or so LEDs on its front panel. Yet it includes a 32-bit microcontroller and runs Linux. -- Jack Klein Home: http://JK-Technology.Com FAQs for comp.lang.c http://c-faq.com/ comp.lang.c++ http://www.parashift.com/c++-faq-lite/ alt.comp.lang.learn.c-c++ http://www.contrib.andrew.cmu.edu/~ajo/docs/FAQ-acllc.html
Hi Rohit:

In a recent post to this group I announced availability of ECI's new
online training web portal free to engineers and others who would like
to learn late-breaking but realistic approaches to embedded device
design.

One of ECI's first three lecture posts answers your question: "How to
use native file shares on one embedded device [such as the one you wish
to design] that actually reside on several other computers and embedded
devices perhaps all with different operating systems and file systems?"

Using software development tools and embedded operating systems such as
from Green Hills or Wind River or an Embedded Linux build such as
LynuxWorks BlueCat Linux all support the type of embedded CIFS file
sharing technique I mention here..

Here is a link to ECI's lecture sponsored by Visuality Systems on the
value of using an embedded device version of  CIFS (the common internet
file system) also known as SMB (shared message blocks) - the network
stack enhancement and popular Windows and Linux Samba standard. With
the CIFS component, an embedded device can share and/or access native
file sharing across multiple devices, computers,  and file systems:

http://www.embeddedcomponents.com/marketplace/makers/visualitynq/intro/



rohit wrote:
> Dear All, > > (Note: I am a newbie to embedded system programming (hardware > programming), so forgive me if it seems a silly query.) > > I need to develop a hardware appliance that will be deployed in the > data path between clients and the storage device (for eg. harddisk). > This hardware box should be able to understand the native storage > protocol (eg. SCSI) over which it is travelling , get the data, perform > some operations on the data (eg. mangle the data) and send forward to > the harddisk. > > So my query is: > 1) How to get started for developing such hardware appliance ? Do we > need to program the logic for it in ASSEMBLY language and burn it into > the eeprom ? Or there are some alternative methods to achieve the same? > > 2) What are the possible tools available for such types of hardware > programming ? > > 3) Basically, I need to embedded the protocol specification into my > hardware. Is that practically feasible ? Or I am going in the wrong > direction. Please guide. > > Any pointers, references, comments will be of great help. > > Thanks and Regards, > Rohit Dhamija
ronf@embeddedcomponents.com wrote:
> Here is a link to ECI's lecture sponsored by Visuality Systems on the > value of using an embedded device version of CIFS (the common internet > file system) also known as SMB (shared message blocks) - the network > stack enhancement and popular Windows and Linux Samba standard. With > the CIFS component, an embedded device can share and/or access native > file sharing across multiple devices, computers, and file systems:
SMB stood for "Server Message Block" not "Shared Message Blocks". Implementing CIFS is a pain as you must agree to onerous terms in order to get the specification from Microsoft. Samba was written by reverse engineering the wire protocol and even today the developers refuse to even look at the MS documentation for fear of IP contamination and future litigation by Microsoft. With regards to the original poster you seem to want to implement a bridge which does ATA-Some processing-ATA or SCSI-SomeProcessing-SCSI. You are unlikely to get any reasonable speed out of this without designing a piece of custom hardware. It would probably be much easier to write some kind of iSCSI proxy software. -p -- "Unix is user friendly, it's just picky about who its friends are." - Anonymous --------------------------------------------------------------------
Oops you're right about SMB acronym Paul - thanks for that.

However, embedded CIFS is already available as a reusable component for
real-time operating systems - that was my point about watching ECI's
lecture introducing this new component as a way to open up native file
shares. I don't mean a re-write from a standards document or a skinning
of Samba. The rewrite could take years and Samba is just too darn large
a foot print. Not many are aware of this complex protocol being
available now as a simple to use, reusable component for client/server
file sharing.

iSCSI could be an interesting approach to consider too - depending on
the design details.

Ron

Paul Gotch wrote:
> ronf@embeddedcomponents.com wrote: > > Here is a link to ECI's lecture sponsored by Visuality Systems on the > > value of using an embedded device version of CIFS (the common internet > > file system) also known as SMB (shared message blocks) - the network > > stack enhancement and popular Windows and Linux Samba standard. With > > the CIFS component, an embedded device can share and/or access native > > file sharing across multiple devices, computers, and file systems: > > SMB stood for "Server Message Block" not "Shared Message Blocks". > Implementing CIFS is a pain as you must agree to onerous terms in order to > get the specification from Microsoft. > > Samba was written by reverse engineering the wire protocol and even today > the developers refuse to even look at the MS documentation for fear of IP > contamination and future litigation by Microsoft. > > With regards to the original poster you seem to want to implement a > bridge which does ATA-Some processing-ATA or SCSI-SomeProcessing-SCSI. You > are unlikely to get any reasonable speed out of this without designing a > piece of custom hardware. > > It would probably be much easier to write some kind of iSCSI proxy software. > > -p > -- > "Unix is user friendly, it's just picky about who its friends are." > - Anonymous > --------------------------------------------------------------------
Hi Jack,

Let me try to answer your queries:
-----
>>Or do you mean that you are trying to do this all in hardware, such as >>in an FPGA?
Comments: Yes, I need to do all this in hardware. I need to develop my own peice of hardware (or get the basic h/w from third party) and embedd the entire logic in it. This is necesssary since I donot want any software client to be installed in the clients or host computers which through which data will be coming/going.
> Is this some sort of data protection tool? Are you trying to encrypt > on a file-by-file basis, or an a low level block basis?
Comments: I am trying to encrypt on low level block basis. My main objective it that irrespective of the data that is travelling through any channel, my hardware should recognize it and perform necessary logic and then send it further.
> Anything that you can program in an assembly language, you can program > in a higher level language.
Comments: For example if I build the entire logic in C language, can it be directly embedded into the hardware that I built ?
> I think that your chances of getting useful answers will improve > greatly if you provide more information about what you are trying to > do. And what the term "hardware appliance" means to you and your > company and your customers. Most embedded systems could be asked as > "hardware appliances", because usually they don't operate with > keyboards, mice, and video displays.
Comments: Please see my first comment. Hardware appliance will be a hardware box that will sit in the network between clients and the storage device. It will not be operated with any keyboard / mice. It will have LED display for its information. For example like a router, my device will also be a hardware I have the concept but I need to know how to get started, eg. What tools are required for programming in such kind of hardware. Any useful information in this regard is most welcome!!! Thanks for your precious time, Regards, Rohit Dhamija Jack Klein wrote:
> On 10 Sep 2006 09:12:29 -0700, "rohit" <rohit.dhamija@gmail.com> wrote > in comp.arch.embedded: > > > Dear All, > > > > (Note: I am a newbie to embedded system programming (hardware > > programming), so forgive me if it seems a silly query.) > > > > I need to develop a hardware appliance that will be deployed in the > > The first question that I need to ask is what is your definition of a > "hardware appliance"? > > Most hardware appliances today have one or more processors in them > (microprocessor, microcontroller, DSP). > > Or do you mean that you are trying to do this all in hardware, such as > in an FPGA? > > > data path between clients and the storage device (for eg. harddisk). > > This hardware box should be able to understand the native storage > > protocol (eg. SCSI) over which it is travelling , get the data, perform > > some operations on the data (eg. mangle the data) and send forward to > > the harddisk. > > Is this some sort of data protection tool? Are you trying to encrypt > on a file-by-file basis, or an a low level block basis? > > > So my query is: > > 1) How to get started for developing such hardware appliance ? Do we > > need to program the logic for it in ASSEMBLY language and burn it into > > the eeprom ? Or there are some alternative methods to achieve the same? > > Anything that you can program in an assembly language, you can program > in a higher level language. > > > 2) What are the possible tools available for such types of hardware > > programming ? > > > > 3) Basically, I need to embedded the protocol specification into my > > hardware. Is that practically feasible ? Or I am going in the wrong > > direction. Please guide. > > > > Any pointers, references, comments will be of great help. > > > > Thanks and Regards, > > Rohit Dhamija > > I think that your chances of getting useful answers will improve > greatly if you provide more information about what you are trying to > do. And what the term "hardware appliance" means to you and your > company and your customers. Most embedded systems could be asked as > "hardware appliances", because usually they don't operate with > keyboards, mice, and video displays. > > There is a router on my desk that connects my computers to a cable > modem. I imaging that most people would classify it as a "hardware > appliance". It's only visible interface is half a dozen or so LEDs on > its front panel. Yet it includes a 32-bit microcontroller and runs > Linux. > > -- > Jack Klein > Home: http://JK-Technology.Com > FAQs for > comp.lang.c http://c-faq.com/ > comp.lang.c++ http://www.parashift.com/c++-faq-lite/ > alt.comp.lang.learn.c-c++ > http://www.contrib.andrew.cmu.edu/~ajo/docs/FAQ-acllc.html
rohit <rohit.dhamija@gmail.com> wrote:

> I need to develop a hardware appliance that will be deployed in the > data path between clients and the storage device (for eg. harddisk).
"Need" to? Says who? And what's keeping you (or them) from just using the encryption that all reasonably modern ATA disks (and SCSI, too, I'll assume) already support?
> This hardware box should be able to understand the native storage > protocol (eg. SCSI) over which it is travelling ,
I take you don't actually plan to try to transport a physical device over a signal line, so let's assum you had said "travelling over it" instead of "over which ...". ;->
> 1) How to get started for developing such hardware appliance ?
With all due respect, if you don't even know how to start, you're far away from being fit for the task. You're years away from a serious chance to succeed.
> 2) What are the possible tools available for such types of hardware > programming ?
Books on gate arrays and ASIC design, and lots of time. Followed by a starter kit, a *much* smaller project to sharpen your tools on, and at least one year to do the job. -- Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de) Even if all the snow were burnt, ashes would remain.
Hi Bernhard,

Thanks for your valuable comments. Please find my comments below:

Query:
> And what's keeping you (or them) from just using the encryption that > all reasonably modern ATA disks (and SCSI, too, I'll assume) already > support?
Comments: Are these encryption mechanism also integrated into Network attached storages (NAS) ? I am targetting SAN/NAS envrionments. Query:
> I take you don't actually plan to try to transport a physical device > over a signal line, so let's assum you had said "travelling over it" > instead of "over which ...". ;->
Comments: Yes, i mean to say the same what you assumed.
> With all due respect, if you don't even know how to start, you're far > away from being fit for the task. You're years away from a serious > chance to succeed.
Comments: Thanks, donot worry I have the time and enough bandwidth to hire the expert resources to work with, if required.
> Books on gate arrays and ASIC design, and lots of time. Followed by a > starter kit, a *much* smaller project to sharpen your tools on, and at > least one year to do the job.
Comments: Thanks for your suggestion. Can you be some more specific ? I have some idea in order to build a sample pilot project for the same, please send your expert comments/suggestions on the same: Step1) The entire logic will be burnt in the hardware. We shall have to take the hardware with embedded OS pre installed in it - Embedded Linux. Step2) So now i can do the programming on a Linux machine and then port (i.e. burn) the same program logic into the hardware. Am I correct in this ? So, I intend to take 3 computer machine having Linux installed on it. 1st computer - will act as a client 2nd computer - will act as myhardware appliance. 2nd computer will be my devlopment machine. Initially it will contain logic to undersand SCSI protocol only. 3rd computer - will act as Storage device. Please send your useful comments suggestions. Thanks and Regards, Rohit Hans-Bernhard Broeker wrote:
> rohit <rohit.dhamija@gmail.com> wrote: > > > I need to develop a hardware appliance that will be deployed in the > > data path between clients and the storage device (for eg. harddisk). > > "Need" to? Says who? > > And what's keeping you (or them) from just using the encryption that > all reasonably modern ATA disks (and SCSI, too, I'll assume) already > support? > > > This hardware box should be able to understand the native storage > > protocol (eg. SCSI) over which it is travelling , > > I take you don't actually plan to try to transport a physical device > over a signal line, so let's assum you had said "travelling over it" > instead of "over which ...". ;-> > > > 1) How to get started for developing such hardware appliance ? > > With all due respect, if you don't even know how to start, you're far > away from being fit for the task. You're years away from a serious > chance to succeed. > > > 2) What are the possible tools available for such types of hardware > > programming ? > > Books on gate arrays and ASIC design, and lots of time. Followed by a > starter kit, a *much* smaller project to sharpen your tools on, and at > least one year to do the job. > > -- > Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de) > Even if all the snow were burnt, ashes would remain.
rohit <rohit.dhamija@gmail.com> wrote:

> Thanks for your valuable comments. Please find my comments below:
Interesting that you didn't bother to answer my first query, about who decreed that this is what you "need" to do.
> Query: > > And what's keeping you (or them) from just using the encryption that > > all reasonably modern ATA disks (and SCSI, too, I'll assume) already > > support? > Comments: > Are these encryption mechanism also integrated into Network attached > storages (NAS) ?
They're in the harddisk themselves. So if the answer isn't yes, that would arguably be major oversight of the controller sitting between the disks and the network.
> Step1) The entire logic will be burnt in the hardware. We shall have > to take the hardware with embedded OS pre installed in it - Embedded > Linux.
You're being seriously unreasonable. You openly admitted that you have no idea what you're doing, and now you're casting your first decision in stone, based on --- well: what?
> Step2) So now i can do the programming on a Linux machine and then > port (i.e. burn) the same program logic into the hardware.
No, you can't. Because the thing you're trying to build won't run Linux. If it ever becomes reality, it'll most probably not even have an ordinary CPU to start with. -- Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de) Even if all the snow were burnt, ashes would remain.
>>You're being seriously unreasonable. You openly admitted that you >>have no idea what you're doing, and now you're casting your first >>decision in stone, based on --- well: what?
I admitted this 3 days back. You think i myself will not work on this idea ??? I thought of some options and put forward the one i think is feasible.
>> No, you can't. Because the thing you're trying to build won't run > Linux.
Why ? Any reason it cannot run on Linux ?? Rohit Hans-Bernhard Broeker wrote:
> rohit <rohit.dhamija@gmail.com> wrote: > > > Thanks for your valuable comments. Please find my comments below: > > Interesting that you didn't bother to answer my first query, about who > decreed that this is what you "need" to do. > > > Query: > > > And what's keeping you (or them) from just using the encryption that > > > all reasonably modern ATA disks (and SCSI, too, I'll assume) already > > > support? > > Comments: > > Are these encryption mechanism also integrated into Network attached > > storages (NAS) ? > > They're in the harddisk themselves. So if the answer isn't yes, that > would arguably be major oversight of the controller sitting between > the disks and the network. > > > Step1) The entire logic will be burnt in the hardware. We shall have > > to take the hardware with embedded OS pre installed in it - Embedded > > Linux. > > You're being seriously unreasonable. You openly admitted that you > have no idea what you're doing, and now you're casting your first > decision in stone, based on --- well: what? > > > Step2) So now i can do the programming on a Linux machine and then > > port (i.e. burn) the same program logic into the hardware. > > No, you can't. Because the thing you're trying to build won't run > Linux. If it ever becomes reality, it'll most probably not even have > an ordinary CPU to start with. > > -- > Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de) > Even if all the snow were burnt, ashes would remain.