This is a multi-part message in MIME format.
--------------070708030608080906050107
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
D Yuniskis wrote:
> Clifford Heath wrote:
>> When I implemented a P2P file distribution platform, I
>> decided that multicast wasn't useful and went instead for
> Why? --------------------^^^^^^^^^^^^^
For this app, the computers targeted are almost always
on the same subnet, so broadcasts naturally get propagated
to just the places they're needed (by default, broadcast
traffic stops at subnet boundaries). LAN traffic is regarded
as essentially free, it's WAN traffic that needs to be
limited and shared.
>> broadcasts. The LDSS protocol (Local Download Sharing
> I'll have to look at the RFC's...
It got to a Draft, which has expired, but I've attached it
below.
Very simple protocol; two messages only (since files are
only identified by SHA-1 hash); just NeedFile and WillSend,
having the same packet structure.
>> normal system operations. It was an interesting project!
> This is geared towards asynchronous sharing, no doubt.
Software distribution. All machines fetch an individual
policy file saying what software they should install, and
in most cases there is overlap; more than one machine
needs the same software. Rather than all downloading a
separate copy, they announce their plans, progress, and
ETA, so others know they can wait for a LAN transfer when
it's done.
> How would you (re)consider your design choices in a
> *synchronous* environment?
In the same-subnet scenario, I'd probably still use broadcast,
unless the utilization is likely to reach a significant
percentage (say, >25%) of the media's capability, or there
is a likelihood of multiple synchronized groups which won't
necessarily have to pass traffic across the same link.
The latter case is pretty rare, actually - a home media subnet
is likely all going through one switch and hence limited by
its capability. Using a IGMP aware router is unlikely to help.
> How would you (re)consider that same scenario in a wireless
> network (with nodes closely located -- "tight weave"
> instead of a "loose mesh")?
I'm not familiar with the implementation of broadcast/multicast
IP in a wireless environment, but I can't imagine that it would
change very much.
Clifford Heath.
--------------070708030608080906050107
Content-Type: text/plain;
name="draft-heath-ldss-00.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="draft-heath-ldss-00.txt"
ManageSoft C. Heath
Draft ManageSoft
February 2006
Local Download Sharing Service.
Copyright Notice
Copyright (C) ManageSoft 2006.
Abstract
This protocol provides file sharing for LAN-based peers on small LANS
(up to twenty nodes) where peers have received a cryptographic digest
of each file they require. It also provides progress monitoring and
download scheduling for files being received from outside the LAN,
allowing single-download of such files.
Search Protocol
The Local Download Sharing Service protocol consists of two messages:
NeedFile and WillSend, broadcast on an agreed UDP port. Since
broadcast messages are normally dropped at the first router, this
protocol is only useful on switched LANs unless special router
configuration is performed.
Either message may be sent at any time, though there is an
expectation that an application that sends a NeedFile message will
see any matching WillSend messages from a capable peer within a
defined search duration, such as twenty seconds.
Search Message Format
Both messages have substantially similar structure:
0 7 8 15 16 23 24 31
+----------+----------+----------+----------+
0 | Message-ID | Digest- | Compres- |
| | type | sion |
+----------+----------+----------+----------+
4 | Cookie |
| |
+----------+----------+----------+----------+
8 | |
| Digest (32) |
......
+----------+----------+----------+----------+
40 | |
| Size |
| |
| |
+----------+----------+----------+----------+
48 | |
| Offset |
| |
| |
+----------+----------+----------+----------+
56 | Target-Time |
| |
+----------+----------+----------+----------+
60 | Unused |
| |
+----------+----------+----------+----------+
64 | Port | Rate |
| | |
+----------+----------+----------+----------+
66 | Priority | Property-Count |
| | |
+----------+----------+----------+----------+
72 | Properties |
.....
+----------+----------+----------+----------+
Message-ID is zero (0) for NeedFile, one (1) for WillSend. Other
values are not defined.
Digest-type indicates what type of cryptographic digest is present in
the digest field. It is set to zero (0) for MD5, one (1) for SHA-1,
and two (2) for SHA-256.
Compression indicates what type of compression is used or expected It
is set to zero (0) for no compression, one (1) for GZIP.
Cookie is a four-byte random value that carries no meaning, but
should be set to a random value with each transmitted message to help
recognised a received message as having originated in the receiving
context.
Digest is a 32-octet field that contains the bytes of the
cryptographic digest, padded with zero octets where the digest is
smaller than 32 bytes.
Size is a 64-bit integer indicating the size of the requested file in
bytes. If the size is not known, this field may be set to zero.
Offset is a 64-bit integer indicating the number of bytes of the
requested file that are already available and either not needed
(NeedFile) or that could be sent immediately (WillSend).
Target-time is a 32-bit integer indicating a time in seconds from the
time the message is transmitted. For a NeedFile request, it
indicates a time by which the requester hopes the requested file
transfer can be completed. For a WillSend message, it indicates a
time by which the sender expects that the entire file will be
available (Offset equals Size). In response to a NeedFile request,
the intention is that the requester should delay further requests for
that file until the specified time, unless another party can provide
the requested file sooner.
The Unused field should be set to zero.
The Port field, if non-zero, indicates a TCP port number on which a
file transfer can take place, as discussed in the section below on
TCP File Transfer protocols.
The Rate field is a 16-bit integer which may be set to a non-zero
value in a WillSend message to indicate the approximate current rate
at which the subject field is being received (the rate at which Avail
is increasing). It's represented as the integer part of the
logarithm to base 2 of the data rate estimate in bytes per second,
multiplied by 2048. This defines the limits on the representation of
data rate: a rate value of 1 corresponds to a data rate of just over
1 byte per second, and a rate value of 65535 corresponds to over
4GB/second. This representation allows a resolution of under 1%.
Finally the Property-Count field contains a count of following
name/value pairs that extend the information in the basic message.
No property names are yet defined, so the Property-Count field must
always be set to zero. When property names are defined in future,
the structure of the following data will be defined.
TCP File Transfer protocols
In the case of a NeedFile message, the Port number indicates that the
sender is listening on that port for the requested TCP byte stream,
which should start at the requested Offset and run for Size-Offset
bytes, or until the file is complete and matches the digest. No
structure is superimposed over such transfers. The first responder
to connect to the specified port should transmit the requested
stream. Other responders (or any response that comes too late, after
the requester has stopped listening), will normally be rejected, or
accepted and dropped without the acceptor reading any bytes.
In the case of a WillSend message, the port number indicates that the
sender is listening for TCP requests on the specified port. Such
requests bear a similarity to simple HTTP requests. They consist of
four lines separated by linefeed (ASCII 10) characters. Each line
consists of a header keyword, a colon and space character, and a
value. The four headers are "Digest", "Start", "Size" and
"Compression". The Digest value is the concatenation of the digest
name ("md5", "sha-1", "sha-256"), a dash character "-", and enough
pairs of hexadecimal digits to form the digest contents. The Start
and Size values are numbers represented as strings of decimal digits.
Start is equivalent to the Offset field in the search protocol.
Compression is a decimal number, 0 for none and 1 for gzip.
When the request has been received, the requested file portion will
normally be transmitted, assuming the offered file is still available
and the sender is not too busy. If any error occurs, the connection
is dropped and the requester should use an alternate source for the
requested file or conduct a fresh search.
Any TCP transfer is subject to early termination by either party,
which is not to be considered an error by the other party.
Progress announcements
Without regard for any NeedFile messages, any program receiving (say
from a slow WAN connection) a file that is available for sharing on
the LAN may send a WillSend message to indicate rate of progress and
expected time to completion. If sent frequently enough in an
environment where all local peers share a limited WAN bandwidth, such
messages may be used to allow each peer to rate-limit its own WAN
usage so as to keep the total usage under some configured limit.
This relies on the rate estimation using a time constant greater than
the interval between such announcements, but in most circumstances
where limited WAN bandwidth is an issue, an interval of beteeen ten
and sixty seconds is appropriate.
Security Considerations
This protocol provides no way to restrict access of certain files
with a limited group of peers. Any file which is available for
sharing may be requested by any peer on the LAN that knows the file's
digest.
Because of the characteristics of the cryptographic digests used,
it's not feasible for a rogue agent to cause bogus files to be
accepted in response to a request, though it may send such a file.
The requester will normally have performed a search and received one
or more offers. If a file turns out not to have the expected digest,
that offer and the bad file is discarded and other offers tried.
Should all offers be bad, the requester will normally resort to
fetching the required file from the WAN, disregarding peers that may
claim to have it. This behaviour limits the extent of a denial of
service attack against the protocol.
Finally, because a requester normally knows the expected size of the
file, any transfer which would continue past that size and possibly
fill up the available storage space can be avoided.
Author's Address:
Clifford J. Heath
ManageSoft,
56-60 Rutland Rd,
Box Hill 3128,
Melbourne, Australia.
--------------070708030608080906050107--